noteThis would be a living document to capture various baselining snapshots while :-
Troubleshooting and applying a patch
Changing Software or Network configurations
Adding new scenarios and changing load share
Source Code: https://github.com/Bahmni/performance-test
This would be a living document to capture various baselining snapshots while :-
Troubleshooting and applying a patch
Changing Software or Network configurations
Adding new scenarios and changing load share
Source Code: https://github.com/Bahmni/performance-test
⭕️ Automation Technology Stack
⭕️ Current Snapshots
HardwarePerformance environment was running on AWS EKS Custer with single node Node (EC2: t3-large) RAM 8GB 2 vCPU 100GB Secondary storage AWS LINUX x86_64
Total 20 application pods in cluster such as openmrs, bahmni-web, postgresql, abdm etc |
Database (AWS RDS service: db.t3.xlarge) RAM 16GB, 4 vCPU (2 core, 2.5 GHz Intel Scalable Processor) 100GB Secondary storage MySQL, max_connections = 1304
SoftwareOpenMRS Tomcat - Server Server version: Apache Tomcat/7.0.94
Server built: Apr 10 2019 16:56:40 UTC
Server number: 7.0.94.0
OS Name: Linux
OS Version: 5.4.204-113.362.amzn2.x86_64
Architecture: amd64
JVM Version: 1.8.0_212-8u212-b01-1~deb9u1-b01
ThreadPool: Max 200, Min 25 (Default server.xml) |
OpenMRS - Heap Initial Heap: 1024 MB Max Heap: 1536 MB
-Xms1024m -Xmx1536m -XX:NewSize=512m -XX:MaxNewSize=512m -XX:MetaspaceSize=256m -XX:MaxMetaspaceSize=1024m -XX:InitialCodeCacheSize=64m -XX:ReservedCodeCacheSize=96m -XX:SurvivorRatio=4 -XX:TargetSurvivorRatio=40 -XX:+UseParNewGC -XX:ParallelGCThreads=2 -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:+CMSCompactWhenClearAllSoftRefs -XX:CMSInitiatingOccupancyFraction=85 -XX:+CMSScavengeBeforeRemark -XX:+UseGCOverheadLimit -XX:+UseStringDeduplication
Openmrs Connection Pooling hibernate.c3p0.max_size=50
hibernate.c3p0.min_size=0
hibernate.c3p0.timeout=100
hibernate.c3p0.max_statements=0
hibernate.c3p0.idle_test_period=3000
hibernate.c3p0.acquire_increment=1 |
|
📗 Current Results - 40 Concurrent Users
noteReport Link: https://bahmni.github.io/performance-test/longduration_report-20230130141239257_40users_24hrs_all_omods_afterhipfix/index.html
Report Observations:
Simulations | Scenario | Load share | Patient Count | Min Time (ms) | 95th Percentile (ms) | 99th Percentile (ms) | Max Time (ms) |
---|
Frontdesk 50% Traffic | New Patient Registration → Start OPD Visit | 40% | 5760 | 152 | 484 | 648 | 1389 |
Existing Patient Search using ID → Start OPD Visit | 30% | 4320 | 54 | 472 | 676 | 1977 |
Existing Patient Search using Name → Start OPD Visit | 20% | 4320 | 119 | 352 | 507 | 1492 |
Upload Patient Document | 10% | 1440 | 142 | 482 | 581 | 1135 |
Doctor 50% Traffic | Doctor Consultation 8 Observations 2 Lab Orders 3 Medication
| 100% | 5760 | 1364 | 4056 | 4531 | 7291 |
HardwarePerformance environment was running on AWS EKS Custer with single node Node (EC2: m5-xlarge) RAM 16GB 4 vCPU 100GB Secondary storage AWS LINUX x86_64
Total 20 application pods in cluster such as openmrs, bahmni-web, postgresql, abdm etc |
Database (AWS RDS service: db.t3.xlarge) RAM 16GB, 4 vCPU (2 core, 2.5 GHz Intel Scalable Processor) 100GB Secondary storage MySQL, max_connections = 1304
SoftwareOpenMRS Tomcat - Server Server version: Apache Tomcat/7.0.94
Server built: Apr 10 2019 16:56:40 UTC
Server number: 7.0.94.0
OS Name: Linux
OS Version: 5.4.204-113.362.amzn2.x86_64
Architecture: amd64
JVM Version: 1.8.0_212-8u212-b01-1~deb9u1-b01
ThreadPool: Max 200, Min 25 (Default server.xml) |
OpenMRS - Heap Initial Heap: 1024 MB Max Heap: 2536 MB
-Xms1024m -Xmx2536m -XX:NewSize=512m -XX:MaxNewSize=512m -XX:MetaspaceSize=256m -XX:MaxMetaspaceSize=1024m -XX:InitialCodeCacheSize=64m -XX:ReservedCodeCacheSize=96m -XX:SurvivorRatio=4 -XX:TargetSurvivorRatio=40 -XX:+UseParNewGC -XX:ParallelGCThreads=2 -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:+CMSCompactWhenClearAllSoftRefs -XX:CMSInitiatingOccupancyFraction=85 -XX:+CMSScavengeBeforeRemark -XX:+UseGCOverheadLimit -XX:+UseStringDeduplication
Openmrs Connection Pooling hibernate.c3p0.max_size=50
hibernate.c3p0.min_size=0
hibernate.c3p0.timeout=100
hibernate.c3p0.max_statements=0
hibernate.c3p0.idle_test_period=3000
hibernate.c3p0.acquire_increment=1 |
|
📗 Current Results - 70 Concurrent Users
noteReport Link: https://bahmni.github.io/performance-test/longduration_report-20230213133118638_70users_24hours_AfterHIPfix_m5xlarge/index.html
Report Observations:
Simulations | Scenario | Load share | Patient Count | Min Time (ms) | 95th Percentile (ms) | 99th Percentile (ms) | Max Time (ms) |
---|
Frontdesk 50% Traffic | New Patient Registration → Start OPD Visit | 40% | 10080 | 130 | 252 | 309 | 674 |
Existing Patient Search using ID → Start OPD Visit | 30% | 7200 | 49 | 305 | 464 | 2660 |
Existing Patient Search using Name → Start OPD Visit | 20% | 7200 | 135 | 278 | 348 | 573 |
Upload Patient Document | 10% | 2160 | 111 | 206 | 253 | 459 |
Doctor 50% Traffic | Doctor Consultation 8 Observations 2 Lab Orders 3 Medication
| 100% | 10080 | 998 | 2331 | 2608 | 4134 |
🔰 Observations:
Note: The tests performed here are of most demand to users using Bahmni lite compared to the real world clinic activities. So it is assumed safe to implement for the suggested concurrent users under a cluster even when the maximum time response numbers for some activities are not optimal at the moment under test.
⭕️ Base Configuration
The below details are historical data observed during each tests with different configurations at the start of performance analysis for Bahmni lite. Kindly view them for data purposes and not as references.
HardwarePerformance environment was running on AWS EKS Custer with single node Node (EC2: m5-xlarge) RAM 16GB 4 vCPU 100GB Secondary storage AWS LINUX x86_64
Total 15 application pods in cluster such as openmrs, bahmni-web, postgresql, rabbitmq etc |
Database (AWS RDS service: db.t3.xlarge) RAM 16GB, 4 vCPU (2 core, 2.5 GHz Intel Scalable Processor) 100GB Secondary storage MySQL, max_connections = 1304
SoftwareOpenMRS Tomcat - Server Server version: Apache Tomcat/7.0.94
Server built: Apr 10 2019 16:56:40 UTC
Server number: 7.0.94.0
OS Name: Linux
OS Version: 5.4.204-113.362.amzn2.x86_64
Architecture: amd64
JVM Version: 1.8.0_212-8u212-b01-1~deb9u1-b01
ThreadPool: Max 200, Min 25 (Default server.xml) |
OpenMRS - Heap Initial Heap: 256 MB Max Heap: 768 MB
-Xms256m -Xmx768m -XX:PermSize=256m -XX:MaxPermSize=512m
Openmrs Connection Pooling hibernate.c3p0.max_size=50
hibernate.c3p0.min_size=0
hibernate.c3p0.timeout=100
hibernate.c3p0.max_statements=0
hibernate.c3p0.idle_test_period=3000
hibernate.c3p0.acquire_increment=1 |
|
📙 10 Concurrent Users
note Report Link: https://bahmni.github.io/performance-test/baseline_report-20220911073147512_10users_30mins_768mb/index.html Report Observations: Simulations | Scenario | Load share | Patient Count | Min Time (ms) | Max Time (ms) |
---|
Frontdesk 50% Traffic | New Patient Registration → Start OPD Visit | 40% | 174 | 100 | 1077 | Existing Patient Search using ID → Start OPD Visit | 30% | 106 | 243 | 1550 | Existing Patient Search using Name → Start OPD Visit | 20% | 107 | 243 | 1437 | Upload Patient Document | 10% | 27 | 228 | 2169 | Doctor 50% Traffic | Doctor Consultation 8 Observations 2 Lab Orders 3 Medication
| 100% | 187 | 185 | 2092 |
|
📙 25 Concurrent Users
noteHeap Size increased to 2GB since the base heap size wasn't enough to support more than 15 concurrent users |
Report Link: https://bahmni.github.io/performance-test/baseline_report-20220912043950582_25users_15mins_2gb/index.html 🔰 Observations: Simulations | Scenario | Load share | Patient Count | Min Time (ms) | Max Time (ms) |
---|
Frontdesk 50% Traffic | New Patient Registration → Start OPD Visit | 40% | 86 | 161 | 1193 | Existing Patient Search using ID → Start OPD Visit | 30% | 74 | 320 | 1113 | Existing Patient Search using Name → Start OPD Visit | 20% | 64 | 310 | 1144 | Upload Patient Document | 10% | 21 | 213 | 843 | Doctor 50% Traffic | Doctor Consultation 8 Observations 2 Lab Orders 3 Medication
| 100% | 107 | 259 | 846 |
|
📙 40 Concurrent Users - Standard Traffic condition
noteNetwork: 50 MBPS Heap Size: 2GB
|
Report Link: https://bahmni.github.io/performance-test/baseline_report-20220912053731908_40users_15mins_2gb/index.html 🔰 Observations: Report Observations: Simulations | Scenario | Load share | Patient Count | Min Time (ms) | Max Time (ms) |
---|
Frontdesk 50% Traffic | New Patient Registration → Start OPD Visit | 40% | 130 | 162 | 1523 | Existing Patient Search using ID → Start OPD Visit | 30% | 109 | 356 | 2921 | Existing Patient Search using Name → Start OPD Visit | 20% | 124 | 337 | 4125 | Upload Patient Document | 10% | 36 | 238 | 579 | Doctor 50% Traffic | Doctor Consultation 8 Observations 2 Lab Orders 3 Medication
| 100% | 142 | 252 | 1289 |
|
📙 70 Concurrent Users - High Traffic condition
note Report Link: https://bahmni.github.io/performance-test/baseline_report-20220912071131867_70users_30mins_2gb/index.html 🔰 Observations: Report Observations: Simulations | Scenario | Load share | Patient Count | Min Time (ms) | Max Time (ms) | Notes |
---|
Frontdesk 50% Traffic | New Patient Registration → Start OPD Visit | 40% | 251 | 198 | 52939 | | Existing Patient Search using ID → Start OPD Visit | 30% | 175 | 1200 | 54692 | | Existing Patient Search using Name → Start OPD Visit | 20% | 162 | 979 | 54510 | | Upload Patient Document | 10% | 62 | 409 | 7848 | | Doctor 50% Traffic | Doctor Consultation 8 Observations 2 Lab Orders 3 Medication
| 100% | 193 | 558 | 44887 | |
|
📙 90 Concurrent Users - Peak Traffic condition
note Report Link: https://bahmni.github.io/performance-test/baseline_report-20220912061644613_90users_30mins_8gb/index.html 🔰 Observations: OpenMRS Pod was consuming 11.5GB Memory and 3.22 CPU Overall Cluster was at 12.5GB RAM and 3.6 CPU More than 50% of the consultation and 95% all of Patient lab order fetch api calls suffered timeout.
RDS showed total of 100+ connection during execution whereas OpenMRS Hibernate connection pool size is 50. Since the Memory and CPU consumption was equally high, we suspect that the application was busy fighting for establishing the connection concurrently. We need to identify a way to override OpenMRS connection pool configuration and rerun the tests. |
Report Observations: Simulations | Scenario | Load share | Patient Count | Min Time (ms) | Max Time (ms) | Notes |
---|
Frontdesk 50% Traffic | New Patient Registration → Start OPD Visit | 40% | 254 | 188 | 55802 | | Existing Patient Search using ID → Start OPD Visit | 30% | 192 | 1031 | 56250 | | Existing Patient Search using Name → Start OPD Visit | 20% | 177 | 651 | 53063 | | Upload Patient Document | 10% | 62 | 545 | 59070 | | Doctor 50% Traffic | Doctor Consultation 8 Observations 2 Lab Orders 3 Medication
| 100% | 200 | 626 | 53271 | |
|
⭕️ Pace Based Framework
The framework is based on iterations per time type volumes i.e. a dedicated pace is set for each persona simulation.
Default JVM Configuration
📙 40 Concurrent Users - Standard Traffic condition
noteDuration: 30 mins Ramp Up: 3 mins
Duration: 30 mins Ramp Up: 3 mins
Report Link: https://bahmni.github.io/performance-test/baseline_report-20220923121606971_40users_30mins_defaultJVM/index.html 🔰 Observations: Report Observations: Simulations | Scenario | Load share | Patient Count | Min Time (ms) | 95th Percentile (ms) | 99th Percentile (ms) | Max Time (ms) |
---|
Frontdesk 50% Traffic | New Patient Registration → Start OPD Visit | 40% | 108 | 88 | 191 | 328 | 378 | Existing Patient Search using ID → Start OPD Visit | 30% | 96 | 243 | 481 | 557 | 571 | Existing Patient Search using Name → Start OPD Visit | 20% | 87 | 243 | 579 | 694 | 749 | Upload Patient Document | 10% | 27 | 164 | 209 | 280 | 304 | Doctor 50% Traffic | Doctor Consultation 8 Observations 2 Lab Orders 3 Medication
| 100% | 120 | 332 | 531 | 610 | 1153 |
|
📙 70 Concurrent Users - High Traffic condition
noteDuration: 30 mins Ramp Up: 3 mins
Duration: 30 mins Ramp Up: 3 mins
Report Link: https://bahmni.github.io/performance-test/baseline_report-20220923133122220_70users_30mins_defaultJVM/index.html 🔰 Observations: Report Observations: Simulations | Scenario | Load share | Patient Count | Min Time (ms) | 95th Percentile (ms) | 99th Percentile (ms) | Max Time (ms) |
---|
Frontdesk 50% Traffic | New Patient Registration → Start OPD Visit | 40% | 202 | 85 | 968 | 1058 | 1120 | Existing Patient Search using ID → Start OPD Visit | 30% | 163 | 232 | 1254 | 1700 | 1860 | Existing Patient Search using Name → Start OPD Visit | 20% | 143 | 241 | 1221 | 1630 | 1866 | Upload Patient Document | 10% | 40 | 144 | 984 | 1281 | 1338 | Doctor 50% Traffic | Doctor Consultation 8 Observations 2 Lab Orders 3 Medication
| 100% | 210 | 329 | 1736 | 2538 | 2654 |
|
📙 90 Concurrent Users - Peak Traffic condition
noteDuration: 30 mins Ramp Up: 3 mins
Duration: 30 mins Ramp Up: 3 mins
Report Link: https://bahmni.github.io/performance-test/baseline_report-20220923141108441_90users_30mins_defaultJVM/index.html 🔰 Observations: Report Observations: Simulations | Scenario | Load share | Patient Count | Min Time (ms) | 95th Percentile (ms) | 99th Percentile (ms) | Max Time (ms) |
---|
Frontdesk 50% Traffic | New Patient Registration → Start OPD Visit | 40% | 274 | 81 | 995 | 1064 | 1843 | Existing Patient Search using ID → Start OPD Visit | 30% | 192 | 250 | 1820 | 2193 | 2410 | Existing Patient Search using Name → Start OPD Visit | 20% | 188 | 239 | 2109 | 2223 | 2465 | Upload Patient Document | 10% | 57 | 167 | 972 | 1366 | 1416 | Doctor 50% Traffic | Doctor Consultation 8 Observations 2 Lab Orders 3 Medication
| 100% | 270 | 339 | 2656 | 3752 | 4149 |
|
Tuned JVM Configuration
The JVM was tuned for better efficiency from the default configuration
Configuration
-Xms768m -Xmx768m -XX:NewSize=384m -XX:MaxNewSize=384m -XX:MetaspaceSize=768m -XX:MaxMetaspaceSize=768m -XX:InitialCodeCacheSize=64m -XX:ReservedCodeCacheSize=96m -XX:SurvivorRatio=16 -XX:TargetSurvivorRatio=50 -XX:MaxTenuringThreshold=15 -XX:+UseParNewGC -XX:ParallelGCThreads=16 -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:+CMSCompactWhenClearAllSoftRefs -XX:CMSInitiatingOccupancyFraction=85 -XX:+CMSScavengeBeforeRemark |
📙 40 Concurrent Users - Standard Traffic condition
noteDuration: 30 mins Ramp Up: 3 mins
Duration: 30 mins Ramp Up: 3 mins
Report Link: https://bahmni.github.io/performance-test/baseline_report-20220927073846406_40users_30mins_tunedJVM/index.html 🔰 Observations: Report Observations: Simulations | Scenario | Load share | Patient Count | Min Time (ms) | 95th Percentile (ms) | 99th Percentile (ms) | Max Time (ms) |
---|
Frontdesk 50% Traffic | New Patient Registration → Start OPD Visit | 40% | 119 | 154 | 294 | 361 | 544 | Existing Patient Search using ID → Start OPD Visit | 30% | 75 | 242 | 408 | 453 | 463 | Existing Patient Search using Name → Start OPD Visit | 20% | 96 | 270 | 612 | 765 | 766 | Upload Patient Document | 10% | 35 | 138 | 348 | 474 | 504 | Doctor 50% Traffic | Doctor Consultation 8 Observations 2 Lab Orders 3 Medication
| 100% | 120 | 490 | 687 | 769 | 778 |
|
📙 70 Concurrent Users - High Traffic condition
noteDuration: 30 mins Ramp Up: 3 mins
Duration: 30 mins Ramp Up: 3 mins
Report Link: https://bahmni.github.io/performance-test/baseline_report-20220927100508288_70users_30mins_tunedJVM/index.html 🔰 Observations: Report Observations: Simulations | Scenario | Load share | Patient Count | Min Time (ms) | 95th Percentile (ms) | 99th Percentile (ms) | Max Time (ms) |
---|
Frontdesk 50% Traffic | New Patient Registration → Start OPD Visit | 40% | 206 | 164 | 366 | 537 | 677 | Existing Patient Search using ID → Start OPD Visit | 30% | 146 | 249 | 661 | 806 | 894 | Existing Patient Search using Name → Start OPD Visit | 20% | 157 | 238 | 654 | 743 | 767 | Upload Patient Document | 10% | 60 | 132 | 600 | 710 | 710 | Doctor 50% Traffic | Doctor Consultation 8 Observations 2 Lab Orders 3 Medication
| 100% | 210 | 497 | 978 | 1075 | 1236 |
|
📙 90 Concurrent Users - Peak Traffic condition
noteDuration: 30 mins Ramp Up: 3 mins
Duration: 30 mins Ramp Up: 3 mins
Report Link: https://bahmni.github.io/performance-test/baseline_report-20220927092902911_90users_30mins_tunedJVM/index.html 🔰 Observations: Report Observations: Simulations | Scenario | Load share | Patient Count | Min Time (ms) | 95th Percentile (ms) | 99th Percentile (ms) | Max Time (ms) |
---|
Frontdesk 50% Traffic | New Patient Registration → Start OPD Visit | 40% | 260 | 153 | 362 | 423 | 526 | Existing Patient Search using ID → Start OPD Visit | 30% | 199 | 268 | 786 | 960 | 1163 | Existing Patient Search using Name → Start OPD Visit | 20% | 197 | 276 | 844 | 952 | 1120 | Upload Patient Document | 10% | 62 | 152 | 537 | 741 | 860 | Doctor 50% Traffic | Doctor Consultation 8 Observations 2 Lab Orders 3 Medication
| 100% | 270 | 511 | 1037 | 1321 | 1551 |
|
⭕️ Scenario based pace framework
The scenario based pace framework enables various pace distribution among different scenarios individually.
Configuration
-Xms768m -Xmx768m -XX:NewSize=384m -XX:MaxNewSize=384m -XX:MetaspaceSize=768m -XX:MaxMetaspaceSize=768m -XX:InitialCodeCacheSize=64m -XX:ReservedCodeCacheSize=96m -XX:SurvivorRatio=16 -XX:TargetSurvivorRatio=50 -XX:MaxTenuringThreshold=15 -XX:+UseParNewGC -XX:ParallelGCThreads=16 -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:+CMSCompactWhenClearAllSoftRefs -XX:CMSInitiatingOccupancyFraction=85 -XX:+CMSScavengeBeforeRemark |
📙 40 Concurrent Users - Standard Traffic condition
📙 70 Concurrent Users - High Traffic condition
📙 90 Concurrent Users - Peak Traffic condition