Page Comparison

This would be a living document to capture various baselining snapshots while :-

Troubleshooting and applying a patch
Changing Software or Network configurations
Adding new scenarios and changing load share

Source Code: https://github.com/Bahmni/performance-test

...

Expand

title	40 Concurrent Users - Configuration

Hardware

Performance environment was running on AWS EKS Custer with single node

Node (EC2: t3-large)

RAM 8GB
2 vCPU
100GB Secondary storage
AWS LINUX x86_64

Info
Total 20 application pods in cluster such as openmrs, bahmni-web, postgresql, abdm etc

Database (AWS RDS service: db.t3.xlarge)

RAM 16GB,
4 vCPU (2 core, 2.5 GHz Intel Scalable Processor)
100GB Secondary storage
MySQL, max_connections = 1304

Software

OpenMRS Tomcat - Server

Code Block

Server version: Apache Tomcat/7.0.94
Server built:   Apr 10 2019 16:56:40 UTC
Server number:  7.0.94.0
OS Name:        Linux
OS Version:     5.4.204-113.362.amzn2.x86_64
Architecture:   amd64
JVM Version:    1.8.0_212-8u212-b01-1~deb9u1-b01
ThreadPool:     Max 200, Min 25 (Default server.xml)

OpenMRS - Heap

Initial Heap: 1024 MB
Max Heap: 1536 MB

-Xms1024m -Xmx1536m -XX:NewSize=512m -XX:MaxNewSize=512m -XX:MetaspaceSize=256m -XX:MaxMetaspaceSize=1024m -XX:InitialCodeCacheSize=64m -XX:ReservedCodeCacheSize=96m -XX:SurvivorRatio=4 -XX:TargetSurvivorRatio=40 -XX:+UseParNewGC -XX:ParallelGCThreads=2 -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:+CMSCompactWhenClearAllSoftRefs -XX:CMSInitiatingOccupancyFraction=85 -XX:+CMSScavengeBeforeRemark -XX:+UseGCOverheadLimit -XX:+UseStringDeduplication

Openmrs Connection Pooling

Code Block
hibernate.c3p0.max_size=50 hibernate.c3p0.min_size=0 hibernate.c3p0.timeout=100 hibernate.c3p0.max_statements=0 hibernate.c3p0.idle_test_period=3000 hibernate.c3p0.acquire_increment=1

...

Simulations	Scenario	Load share	Patient Count	Min Time (ms)	95th Percentile (ms)	99th Percentile (ms)	Max Time (ms)
Frontdesk 50% Traffic	New Patient Registration → Start OPD Visit	40%	5760	152	484	648	1389
	Existing Patient Search using ID → Start OPD Visit	30%	4320	54	472	676	1977
	Existing Patient Search using Name → Start OPD Visit	20%	4320	119	352	507	1492
	Upload Patient Document	10%	1440	142	482	581	1135
Doctor 50% Traffic	Doctor Consultation 8 Observations 2 Lab Orders 3 Medication	100%	5760	1364	4056	4531	7291

Expand

title	70 Concurrent Users - Configuration

Hardware

Performance environment was running on AWS EKS Custer with single node

Node (EC2: m5-xlarge)

RAM 16GB
4 vCPU
100GB Secondary storage
AWS LINUX x86_64

Info
Total 20 application pods in cluster such as openmrs, bahmni-web, postgresql, abdm etc

Database (AWS RDS service: db.t3.xlarge)

RAM 16GB,
4 vCPU (2 core, 2.5 GHz Intel Scalable Processor)
100GB Secondary storage
MySQL, max_connections = 1304

Software

OpenMRS Tomcat - Server

Code Block

Server version: Apache Tomcat/7.0.94
Server built:   Apr 10 2019 16:56:40 UTC
Server number:  7.0.94.0
OS Name:        Linux
OS Version:     5.4.204-113.362.amzn2.x86_64
Architecture:   amd64
JVM Version:    1.8.0_212-8u212-b01-1~deb9u1-b01
ThreadPool:     Max 200, Min 25 (Default server.xml)

OpenMRS - Heap

Initial Heap: 1024 MB
Max Heap: 2536 MB

-Xms1024m -Xmx2536m -XX:NewSize=512m -XX:MaxNewSize=512m -XX:MetaspaceSize=256m -XX:MaxMetaspaceSize=1024m -XX:InitialCodeCacheSize=64m -XX:ReservedCodeCacheSize=96m -XX:SurvivorRatio=4 -XX:TargetSurvivorRatio=40 -XX:+UseParNewGC -XX:ParallelGCThreads=2 -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:+CMSCompactWhenClearAllSoftRefs -XX:CMSInitiatingOccupancyFraction=85 -XX:+CMSScavengeBeforeRemark -XX:+UseGCOverheadLimit -XX:+UseStringDeduplication

Openmrs Connection Pooling

Code Block
hibernate.c3p0.max_size=50 hibernate.c3p0.min_size=0 hibernate.c3p0.timeout=100 hibernate.c3p0.max_statements=0 hibernate.c3p0.idle_test_period=3000 hibernate.c3p0.acquire_increment=1

...

Simulations	Scenario	Load share	Patient Count	Min Time (ms)	95th Percentile (ms)	99th Percentile (ms)	Max Time (ms)
Frontdesk 50% Traffic	New Patient Registration → Start OPD Visit	40%	10080	130	252	309	674
	Existing Patient Search using ID → Start OPD Visit	30%	7200	49	305	464	2660
	Existing Patient Search using Name → Start OPD Visit	20%	7200	135	278	348	573
	Upload Patient Document	10%	2160	111	206	253	459
Doctor 50% Traffic	Doctor Consultation 8 Observations 2 Lab Orders 3 Medication	100%	10080	998	2331	2608	4134

...

Expand

title	Details

Hardware

Performance environment was running on AWS EKS Custer with single node

Node (EC2: m5-xlarge)

RAM 16GB
4 vCPU
100GB Secondary storage
AWS LINUX x86_64

Info
Total 15 application pods in cluster such as openmrs, bahmni-web, postgresql, rabbitmq etc

Database (AWS RDS service: db.t3.xlarge)

RAM 16GB,
4 vCPU (2 core, 2.5 GHz Intel Scalable Processor)
100GB Secondary storage
MySQL, max_connections = 1304

Software

OpenMRS Tomcat - Server

Code Block

Server version: Apache Tomcat/7.0.94
Server built:   Apr 10 2019 16:56:40 UTC
Server number:  7.0.94.0
OS Name:        Linux
OS Version:     5.4.204-113.362.amzn2.x86_64
Architecture:   amd64
JVM Version:    1.8.0_212-8u212-b01-1~deb9u1-b01
ThreadPool:     Max 200, Min 25 (Default server.xml)

OpenMRS - Heap

Initial Heap: 256 MB
Max Heap: 768 MB

-Xms256m -Xmx768m -XX:PermSize=256m -XX:MaxPermSize=512m

Openmrs Connection Pooling

Code Block
hibernate.c3p0.max_size=50 hibernate.c3p0.min_size=0 hibernate.c3p0.timeout=100 hibernate.c3p0.max_statements=0 hibernate.c3p0.idle_test_period=3000 hibernate.c3p0.acquire_increment=1

...

Expand

title	Details

Network: 30 MBPS
Duration: 30 mins
Ramp Up: 3 mins
Database pre-state: 2104 Active Patients with 9288 Observations

Report Link: https://bahmni.github.io/performance-test/baseline_report-20220911073147512_10users_30mins_768mb/index.html

Report Observations:

Simulations	Scenario	Load share	Patient Count	Min Time (ms)	Max Time (ms)
Frontdesk 50% Traffic	New Patient Registration → Start OPD Visit	40%	174	100	1077
	Existing Patient Search using ID → Start OPD Visit	30%	106	243	1550
	Existing Patient Search using Name → Start OPD Visit	20%	107	243	1437
	Upload Patient Document	10%	27	228	2169
Doctor 50% Traffic	Doctor Consultation 8 Observations 2 Lab Orders 3 Medication	100%	187	185	2092

...

Expand

title	Details

Network: 30 MBPS
Duration: 15 mins
Ramp Up: 1.5 mins
Database pre-state: 2278 Active Patients with 10720 Observations

Note
Heap Size increased to 2GB since the base heap size wasn't enough to support more than 15 concurrent users

Report Link: https://bahmni.github.io/performance-test/baseline_report-20220912043950582_25users_15mins_2gb/index.html

🔰 Observations:

Simulations	Scenario	Load share	Patient Count	Min Time (ms)	Max Time (ms)
Frontdesk 50% Traffic	New Patient Registration → Start OPD Visit	40%	86	161	1193
	Existing Patient Search using ID → Start OPD Visit	30%	74	320	1113
	Existing Patient Search using Name → Start OPD Visit	20%	64	310	1144
	Upload Patient Document	10%	21	213	843
Doctor 50% Traffic	Doctor Consultation 8 Observations 2 Lab Orders 3 Medication	100%	107	259	846

...

Expand

title	Details

Duration: 15 mins
Ramp Up: 1.5 mins
Database pre-state: 2546 Active Patients with 12303 Observations

Note
Network: 50 MBPS Heap Size: 2GB

Report Link: https://bahmni.github.io/performance-test/baseline_report-20220912053731908_40users_15mins_2gb/index.html

🔰 Observations:

OpenMRS Pod was consuming 4.15GB Memory and 2.59 CPU
Overall Cluster was at 5.2GB RAM and 2.9 CPU

Report Observations:

Simulations	Scenario	Load share	Patient Count	Min Time (ms)	Max Time (ms)
Frontdesk 50% Traffic	New Patient Registration → Start OPD Visit	40%	130	162	1523
	Existing Patient Search using ID → Start OPD Visit	30%	109	356	2921
	Existing Patient Search using Name → Start OPD Visit	20%	124	337	4125
	Upload Patient Document	10%	36	238	579
Doctor 50% Traffic	Doctor Consultation 8 Observations 2 Lab Orders 3 Medication	100%	142	252	1289

...

Expand

title	Details

Duration: 30 mins
Ramp Up: 3 mins
Database pre-state: 2921 Active Patients with 12303 Observations

Note
Network speed was reduced to 14 MBPS to identify any performance patterns compared to 90 concurrent users on better network bandwidth Heap was kept to 8GB

Report Link: https://bahmni.github.io/performance-test/baseline_report-20220912071131867_70users_30mins_2gb/index.html

🔰 Observations:

OpenMRS Cluster Memory and CPU consumption was similar to 90 concurrent user run (Peak traffic)
The API failure rates were very similar to peak traffic

Report Observations:

Simulations	Scenario	Load share	Patient Count	Min Time (ms)	Max Time (ms)	Notes
Frontdesk 50% Traffic	New Patient Registration → Start OPD Visit	40%	251	198	52939
	Existing Patient Search using ID → Start OPD Visit	30%	175	1200	54692
	Existing Patient Search using Name → Start OPD Visit	20%	162	979	54510
	Upload Patient Document	10%	62	409	7848
Doctor 50% Traffic	Doctor Consultation 8 Observations 2 Lab Orders 3 Medication	100%	193	558	44887	Get Patient Lab Orders: 95% KO Save Consultation: 52% KO Get Visits: 18% KO

📙 90 Concurrent Users - Peak Traffic condition

Expand

title	Details

Duration: 30 mins
Ramp Up: 3 mins
Database pre-state: 2676 Active Patients with 13057 Observations

Note
Heap Size was increased to 8GB since 2GB Heap wasnt enough to support more than 40-50 concurrent users Network was bumped up to 50 MBPS

Report Link: https://bahmni.github.io/performance-test/baseline_report-20220912061644613_90users_30mins_8gb/index.html

🔰 Observations:

OpenMRS Pod was consuming 11.5GB Memory and 3.22 CPU
Overall Cluster was at 12.5GB RAM and 3.6 CPU
More than 50% of the consultation and 95% all of Patient lab order fetch api calls suffered timeout.

Info
RDS showed total of 100+ connection during execution whereas OpenMRS Hibernate connection pool size is 50. Since the Memory and CPU consumption was equally high, we suspect that the application was busy fighting for establishing the connection concurrently. We need to identify a way to override OpenMRS connection pool configuration and rerun the tests.

Report Observations:

Simulations	Scenario	Load share	Patient Count	Min Time (ms)	Max Time (ms)	Notes
Frontdesk 50% Traffic	New Patient Registration → Start OPD Visit	40%	254	188	55802
	Existing Patient Search using ID → Start OPD Visit	30%	192	1031	56250
	Existing Patient Search using Name → Start OPD Visit	20%	177	651	53063
	Upload Patient Document	10%	62	545	59070
Doctor 50% Traffic	Doctor Consultation 8 Observations 2 Lab Orders 3 Medication	100%	200	626	53271	Get Patient Lab Orders : 95% KO Save Consultation: with 56% KO Get Visits: 15% KO

⭕️ Pace Based Framework

The framework is based on iterations per time type volumes i.e. a dedicated pace is set for each persona simulation.

⚫ Default JVM Configuration

📙 40 Concurrent Users - Standard Traffic condition

Expand

title	Details

Duration: 30 mins
Ramp Up: 3 mins

Note
Network: 60 MBPS JVM: “-Dfile.encoding=UTF-8 -server -Xms256m -Xmx768m -XX:PermSize=256m -XX:MaxPermSize=512m”

Report Link: https://bahmni.github.io/performance-test/baseline_report-20220923121606971_40users_30mins_defaultJVM/index.html

🔰 Observations:

OpenMRS Pod was consuming GB Memory and CPU
Overall Cluster was at GB RAM and CPU

Report Observations:

Simulations	Scenario	Load share	Patient Count	Min Time (ms)	95th Percentile (ms)	99th Percentile (ms)	Max Time (ms)
Frontdesk 50% Traffic	New Patient Registration → Start OPD Visit	40%	108	88	191	328	378
	Existing Patient Search using ID → Start OPD Visit	30%	96	243	481	557	571
	Existing Patient Search using Name → Start OPD Visit	20%	87	243	579	694	749
	Upload Patient Document	10%	27	164	209	280	304
Doctor 50% Traffic	Doctor Consultation 8 Observations 2 Lab Orders 3 Medication	100%	120	332	531	610	1153

...

Expand

title	Details

Duration: 30 mins
Ramp Up: 3 mins

Note
Network: 60 MBPS JVM: “-Dfile.encoding=UTF-8 -server -Xms256m -Xmx768m -XX:PermSize=256m -XX:MaxPermSize=512m”

Report Link: https://bahmni.github.io/performance-test/baseline_report-20220923133122220_70users_30mins_defaultJVM/index.html

🔰 Observations:

OpenMRS Pod was consuming GB Memory and CPU
Overall Cluster was at GB RAM and CPU

Report Observations:

Simulations	Scenario	Load share	Patient Count	Min Time (ms)	95th Percentile (ms)	99th Percentile (ms)	Max Time (ms)
Frontdesk 50% Traffic	New Patient Registration → Start OPD Visit	40%	202	85	968	1058	1120
	Existing Patient Search using ID → Start OPD Visit	30%	163	232	1254	1700	1860
	Existing Patient Search using Name → Start OPD Visit	20%	143	241	1221	1630	1866
	Upload Patient Document	10%	40	144	984	1281	1338
Doctor 50% Traffic	Doctor Consultation 8 Observations 2 Lab Orders 3 Medication	100%	210	329	1736	2538	2654

...

Expand

title	Details

Duration: 30 mins
Ramp Up: 3 mins

Note
Network: 60 MBPS JVM: “-Dfile.encoding=UTF-8 -server -Xms256m -Xmx768m -XX:PermSize=256m -XX:MaxPermSize=512m”

Report Link: https://bahmni.github.io/performance-test/baseline_report-20220923141108441_90users_30mins_defaultJVM/index.html

🔰 Observations:

OpenMRS Pod was consuming GB Memory and CPU
Overall Cluster was at GB RAM and CPU

Report Observations:

Simulations	Scenario	Load share	Patient Count	Min Time (ms)	95th Percentile (ms)	99th Percentile (ms)	Max Time (ms)
Frontdesk 50% Traffic	New Patient Registration → Start OPD Visit	40%	274	81	995	1064	1843
	Existing Patient Search using ID → Start OPD Visit	30%	192	250	1820	2193	2410
	Existing Patient Search using Name → Start OPD Visit	20%	188	239	2109	2223	2465
	Upload Patient Document	10%	57	167	972	1366	1416
Doctor 50% Traffic	Doctor Consultation 8 Observations 2 Lab Orders 3 Medication	100%	270	339	2656	3752	4149

⚫ Tuned JVM Configuration

The JVM was tuned for better efficiency from the default configuration

...

Expand

title	Details

Duration: 30 mins
Ramp Up: 3 mins

Note
Network: 60 MBPS

Report Link: https://bahmni.github.io/performance-test/baseline_report-20220927073846406_40users_30mins_tunedJVM/index.html

🔰 Observations:

OpenMRS Pod was consuming 2.75 GB Memory and 0.411 CP

Report Observations:

Simulations	Scenario	Load share	Patient Count	Min Time (ms)	95th Percentile (ms)	99th Percentile (ms)	Max Time (ms)
Frontdesk 50% Traffic	New Patient Registration → Start OPD Visit	40%	119	154	294	361	544
	Existing Patient Search using ID → Start OPD Visit	30%	75	242	408	453	463
	Existing Patient Search using Name → Start OPD Visit	20%	96	270	612	765	766
	Upload Patient Document	10%	35	138	348	474	504
Doctor 50% Traffic	Doctor Consultation 8 Observations 2 Lab Orders 3 Medication	100%	120	490	687	769	778

...

Expand

title	Details

Duration: 30 mins
Ramp Up: 3 mins

Note
Network: 60 MBPS

Report Link: https://bahmni.github.io/performance-test/baseline_report-20220927100508288_70users_30mins_tunedJVM/index.html

🔰 Observations:

OpenMRS Pod was consuming 2.77 GB Memory and 1.22 CPU

Report Observations:

Simulations	Scenario	Load share	Patient Count	Min Time (ms)	95th Percentile (ms)	99th Percentile (ms)	Max Time (ms)
Frontdesk 50% Traffic	New Patient Registration → Start OPD Visit	40%	206	164	366	537	677
	Existing Patient Search using ID → Start OPD Visit	30%	146	249	661	806	894
	Existing Patient Search using Name → Start OPD Visit	20%	157	238	654	743	767
	Upload Patient Document	10%	60	132	600	710	710
Doctor 50% Traffic	Doctor Consultation 8 Observations 2 Lab Orders 3 Medication	100%	210	497	978	1075	1236

...

Expand

title	Details

Duration: 30 mins
Ramp Up: 3 mins

Note
Network: 60 MBPS

Report Link: https://bahmni.github.io/performance-test/baseline_report-20220927092902911_90users_30mins_tunedJVM/index.html

🔰 Observations:

OpenMRS Pod was consuming 2.76 GB Memory and 1.42 CPU

Report Observations:

Simulations	Scenario	Load share	Patient Count	Min Time (ms)	95th Percentile (ms)	99th Percentile (ms)	Max Time (ms)
Frontdesk 50% Traffic	New Patient Registration → Start OPD Visit	40%	260	153	362	423	526
	Existing Patient Search using ID → Start OPD Visit	30%	199	268	786	960	1163
	Existing Patient Search using Name → Start OPD Visit	20%	197	276	844	952	1120
	Upload Patient Document	10%	62	152	537	741	860
Doctor 50% Traffic	Doctor Consultation 8 Observations 2 Lab Orders 3 Medication	100%	270	511	1037	1321	1551

...

Versions Compared

Old Version 37

New Version Current

Key

Hardware

Software

Simulations

Scenario

Load share

Patient Count

Min Time (ms)

95th Percentile (ms)

99th Percentile (ms)

Max Time (ms)

Hardware

Software

Simulations

Scenario

Load share

Patient Count

Min Time (ms)

95th Percentile (ms)

99th Percentile (ms)

Max Time (ms)

Hardware

Software

📙 90 Concurrent Users - Peak Traffic condition

⭕️ Pace Based Framework

⚫ Default JVM Configuration

📙 40 Concurrent Users - Standard Traffic condition

⚫ Tuned JVM Configuration