Agile software development is one of the methodologies used by organizations to manage software development projects. The method allows processes to adapt quickly and respond to customer expectations. Nevertheless, agile method utilization may involve challenges in software development projects. It is crucial to identify the relevant challenges of agile software development to attain project success. Extant literature has limited examination concerning project setting and the categorization of agile challenges. This study explores agile challenges in the external software development project and expert perspectives for possible categorization. This study aims to map agile software development challenges using a systematic literature review (SLR) and the Delphi method to address the gap. Nineteen challenges and five categories are identified from prior studies using the SLR. The Delphi method's result suggests five categories of challenges and improves the challenge concepts from previous studies. Finally, this study can serve as a starting point to examine which group of challenges substantially impacts in information system industry.