Improvement of the DDPG Algorithm via Twin Delayed DDPG (TD3) on Vertical Rocket Landing Control System

Faisal Amir Maz; Prawito Prajitno; Rika Andiarti; Rini Akmeliawati; Djati Handoko; Sastra Kusuma Wijaya; Larasmoyo Nugroho

doi:10.1063/5.0181658

Improvement of the DDPG Algorithm via Twin Delayed DDPG (TD3) on Vertical Rocket Landing Control System

Faisal Amir Maz, Prawito Prajitno, Rika Andiarti, Rini Akmeliawati, Djati Handoko, Sastra Kusuma Wijaya, Larasmoyo Nugroho

Department of Physics

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

Abstract

The Double Deep Policy Gradient or DDPG Reinforcement Learning algorithm has a tendency to overestimate computations, which results in learning policies that are not optimum. When DDPG agent is applied in the rocket landing control system, high frequency chattering occurs at the main engine's control command and this gives substantial drawbacks for the liquid propulsion life longevity. The Twin-Delayed Deep Deterministic Policy Gradient (TD3) Reinforcement Learning agent is proposed to overcame this chattering by delaying the landing rocket control command, as simulated in two dimensions. This TD3 controller is shown to be able to dampen the engine command as its achievements are contrasted with the outcomes of the DDPG one.

Original language	English
Title of host publication	AIP Conference Proceedings
Editors	Harry Septanto, Muhammad Ilham Adhynugraha, Yenni Vetrita, Cahya Edi Santosa, Peberlin Parulian Sitompul, Erma Yulihastin, Johan Muhamad, Mardianis, Ery Fitrianingsih, Mario Batubara, Prayitno Abadi, Afni Restasari
Publisher	American Institute of Physics Inc.
Edition	1
ISBN (Electronic)	9780735447554
DOIs	https://doi.org/10.1063/5.0181658
Publication status	Published - 11 Dec 2023
Event	9th International Seminar on Aerospace Science and Technology, ISAST 2022 - Virtual, Online, Indonesia Duration: 22 Nov 2022 → 23 Nov 2022

Publication series

Name	AIP Conference Proceedings
Number	1
Volume	2941
ISSN (Print)	0094-243X
ISSN (Electronic)	1551-7616

Conference

Conference	9th International Seminar on Aerospace Science and Technology, ISAST 2022
Country/Territory	Indonesia
City	Virtual, Online
Period	22/11/22 → 23/11/22

Access to Document

10.1063/5.0181658

Cite this

Maz, F. A., Prajitno, P., Andiarti, R., Akmeliawati, R., Handoko, D., Wijaya, S. K., & Nugroho, L. (2023). Improvement of the DDPG Algorithm via Twin Delayed DDPG (TD3) on Vertical Rocket Landing Control System. In H. Septanto, M. I. Adhynugraha, Y. Vetrita, C. E. Santosa, P. P. Sitompul, E. Yulihastin, J. Muhamad, Mardianis, E. Fitrianingsih, M. Batubara, P. Abadi, & A. Restasari (Eds.), AIP Conference Proceedings (1 ed.). Article 020023 (AIP Conference Proceedings; Vol. 2941, No. 1). American Institute of Physics Inc.. https://doi.org/10.1063/5.0181658

Maz, Faisal Amir ; Prajitno, Prawito ; Andiarti, Rika et al. / Improvement of the DDPG Algorithm via Twin Delayed DDPG (TD3) on Vertical Rocket Landing Control System. AIP Conference Proceedings. editor / Harry Septanto ; Muhammad Ilham Adhynugraha ; Yenni Vetrita ; Cahya Edi Santosa ; Peberlin Parulian Sitompul ; Erma Yulihastin ; Johan Muhamad ; Mardianis ; Ery Fitrianingsih ; Mario Batubara ; Prayitno Abadi ; Afni Restasari. 1. ed. American Institute of Physics Inc., 2023. (AIP Conference Proceedings; 1).

@inproceedings{14f217cea4034940b5a6355bdd2657a9,

title = "Improvement of the DDPG Algorithm via Twin Delayed DDPG (TD3) on Vertical Rocket Landing Control System",

abstract = "The Double Deep Policy Gradient or DDPG Reinforcement Learning algorithm has a tendency to overestimate computations, which results in learning policies that are not optimum. When DDPG agent is applied in the rocket landing control system, high frequency chattering occurs at the main engine's control command and this gives substantial drawbacks for the liquid propulsion life longevity. The Twin-Delayed Deep Deterministic Policy Gradient (TD3) Reinforcement Learning agent is proposed to overcame this chattering by delaying the landing rocket control command, as simulated in two dimensions. This TD3 controller is shown to be able to dampen the engine command as its achievements are contrasted with the outcomes of the DDPG one.",

author = "Maz, {Faisal Amir} and Prawito Prajitno and Rika Andiarti and Rini Akmeliawati and Djati Handoko and Wijaya, {Sastra Kusuma} and Larasmoyo Nugroho",

note = "Publisher Copyright: {\textcopyright} 2023 American Institute of Physics Inc.. All rights reserved.; 9th International Seminar on Aerospace Science and Technology, ISAST 2022 ; Conference date: 22-11-2022 Through 23-11-2022",

year = "2023",

month = dec,

day = "11",

doi = "10.1063/5.0181658",

language = "English",

series = "AIP Conference Proceedings",

publisher = "American Institute of Physics Inc.",

number = "1",

editor = "Harry Septanto and Adhynugraha, {Muhammad Ilham} and Yenni Vetrita and Santosa, {Cahya Edi} and Sitompul, {Peberlin Parulian} and Erma Yulihastin and Johan Muhamad and Mardianis and Ery Fitrianingsih and Mario Batubara and Prayitno Abadi and Afni Restasari",

booktitle = "AIP Conference Proceedings",

address = "United States",

edition = "1",

}

Maz, FA, Prajitno, P, Andiarti, R, Akmeliawati, R, Handoko, D, Wijaya, SK & Nugroho, L 2023, Improvement of the DDPG Algorithm via Twin Delayed DDPG (TD3) on Vertical Rocket Landing Control System. in H Septanto, MI Adhynugraha, Y Vetrita, CE Santosa, PP Sitompul, E Yulihastin, J Muhamad, Mardianis, E Fitrianingsih, M Batubara, P Abadi & A Restasari (eds), AIP Conference Proceedings. 1 edn, 020023, AIP Conference Proceedings, no. 1, vol. 2941, American Institute of Physics Inc., 9th International Seminar on Aerospace Science and Technology, ISAST 2022, Virtual, Online, Indonesia, 22/11/22. https://doi.org/10.1063/5.0181658

Improvement of the DDPG Algorithm via Twin Delayed DDPG (TD3) on Vertical Rocket Landing Control System. / Maz, Faisal Amir; Prajitno, Prawito; Andiarti, Rika et al.
AIP Conference Proceedings. ed. / Harry Septanto; Muhammad Ilham Adhynugraha; Yenni Vetrita; Cahya Edi Santosa; Peberlin Parulian Sitompul; Erma Yulihastin; Johan Muhamad; Mardianis; Ery Fitrianingsih; Mario Batubara; Prayitno Abadi; Afni Restasari. 1. ed. American Institute of Physics Inc., 2023. 020023 (AIP Conference Proceedings; Vol. 2941, No. 1).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Improvement of the DDPG Algorithm via Twin Delayed DDPG (TD3) on Vertical Rocket Landing Control System

AU - Maz, Faisal Amir

AU - Prajitno, Prawito

AU - Andiarti, Rika

AU - Akmeliawati, Rini

AU - Handoko, Djati

AU - Wijaya, Sastra Kusuma

AU - Nugroho, Larasmoyo

PY - 2023/12/11

Y1 - 2023/12/11

N2 - The Double Deep Policy Gradient or DDPG Reinforcement Learning algorithm has a tendency to overestimate computations, which results in learning policies that are not optimum. When DDPG agent is applied in the rocket landing control system, high frequency chattering occurs at the main engine's control command and this gives substantial drawbacks for the liquid propulsion life longevity. The Twin-Delayed Deep Deterministic Policy Gradient (TD3) Reinforcement Learning agent is proposed to overcame this chattering by delaying the landing rocket control command, as simulated in two dimensions. This TD3 controller is shown to be able to dampen the engine command as its achievements are contrasted with the outcomes of the DDPG one.

AB - The Double Deep Policy Gradient or DDPG Reinforcement Learning algorithm has a tendency to overestimate computations, which results in learning policies that are not optimum. When DDPG agent is applied in the rocket landing control system, high frequency chattering occurs at the main engine's control command and this gives substantial drawbacks for the liquid propulsion life longevity. The Twin-Delayed Deep Deterministic Policy Gradient (TD3) Reinforcement Learning agent is proposed to overcame this chattering by delaying the landing rocket control command, as simulated in two dimensions. This TD3 controller is shown to be able to dampen the engine command as its achievements are contrasted with the outcomes of the DDPG one.

UR - http://www.scopus.com/inward/record.url?scp=85180388213&partnerID=8YFLogxK

U2 - 10.1063/5.0181658

DO - 10.1063/5.0181658

M3 - Conference contribution

AN - SCOPUS:85180388213

T3 - AIP Conference Proceedings

BT - AIP Conference Proceedings

A2 - Septanto, Harry

A2 - Adhynugraha, Muhammad Ilham

A2 - Vetrita, Yenni

A2 - Santosa, Cahya Edi

A2 - Sitompul, Peberlin Parulian

A2 - Yulihastin, Erma

A2 - Muhamad, Johan

A2 - Mardianis, null

A2 - Fitrianingsih, Ery

A2 - Batubara, Mario

A2 - Abadi, Prayitno

A2 - Restasari, Afni

PB - American Institute of Physics Inc.

T2 - 9th International Seminar on Aerospace Science and Technology, ISAST 2022

Y2 - 22 November 2022 through 23 November 2022

ER -

Maz FA, Prajitno P, Andiarti R, Akmeliawati R, Handoko D, Wijaya SK et al. Improvement of the DDPG Algorithm via Twin Delayed DDPG (TD3) on Vertical Rocket Landing Control System. In Septanto H, Adhynugraha MI, Vetrita Y, Santosa CE, Sitompul PP, Yulihastin E, Muhamad J, Mardianis, Fitrianingsih E, Batubara M, Abadi P, Restasari A, editors, AIP Conference Proceedings. 1 ed. American Institute of Physics Inc. 2023. 020023. (AIP Conference Proceedings; 1). doi: 10.1063/5.0181658

Improvement of the DDPG Algorithm via Twin Delayed DDPG (TD3) on Vertical Rocket Landing Control System

Abstract

Publication series

Conference

Access to Document

Other files and links

Fingerprint

Cite this