ResBench/evaluate/solution_pass_analysis.csv

1		gpt-3.5-turbo	gpt-4	gpt-4o	gpt-o1-mini	llama3.1-405B	qwen-max	qwen-plus	qwen2.5-coder-32B-instruct	codestral
2	Combinational Logic	112 | 5 | 3	117 | 3 | 0	120 | 0 | 0	118 | 1 | 1	115 | 2 | 3	117 | 2 | 1	109 | 1 | 10	112 | 2 | 6	120 | 0 | 0
3	Finite State Machines	23 | 15 | 22	32 | 22 | 6	31 | 24 | 5	39 | 18 | 3	31 | 24 | 5	34 | 26 | 0	27 | 23 | 10	39 | 10 | 11	36 | 6 | 18
4	Mathematical Functions	13 | 19 | 43	6 | 39 | 30	36 | 10 | 29	46 | 24 | 5	7 | 6 | 62	26 | 27 | 22	20 | 26 | 29	5 | 8 | 62	0 | 3 | 72
5	Basic Arithmetic Operations	37 | 2 | 36	63 | 8 | 4	66 | 9 | 0	68 | 4 | 3	43 | 2 | 30	38 | 22 | 15	27 | 13 | 35	54 | 6 | 15	62 | 13 | 0
6	Bitwise and Logical Operations	35 | 0 | 25	55 | 0 | 5	58 | 2 | 0	59 | 0 | 1	52 | 0 | 8	47 | 0 | 13	33 | 11 | 16	36 | 0 | 24	55 | 0 | 5
7	Pipelining	0 | 59 | 16	11 | 54 | 10	26 | 49 | 0	15 | 38 | 22	7 | 38 | 30	15 | 32 | 28	16 | 26 | 33	21 | 31 | 23	6 | 56 | 13
8	Polynomial Evaluation	19 | 3 | 53	69 | 0 | 6	74 | 1 | 0	68 | 5 | 2	58 | 6 | 11	55 | 2 | 18	28 | 5 | 42	65 | 7 | 3	69 | 6 | 0
9	Machine Learning	31 | 3 | 41	60 | 8 | 7	60 | 13 | 2	73 | 1 | 1	45 | 28 | 2	63 | 12 | 0	61 | 12 | 2	57 | 2 | 16	64 | 8 | 3
10	Financial Computing	9 | 23 | 28	21 | 22 | 17	29 | 13 | 18	20 | 20 | 20	11 | 21 | 28	28 | 15 | 17	15 | 12 | 33	16 | 7 | 37	17 | 23 | 20
11	Encryption	30 | 0 | 15	30 | 2 | 13	25 | 20 | 0	30 | 0 | 15	26 | 0 | 19	25 | 9 | 11	30 | 1 | 14	30 | 0 | 15	30 | 0 | 15
12	Physics	45 | 3 | 12	57 | 0 | 3	53 | 4 | 3	54 | 5 | 1	41 | 11 | 8	49 | 7 | 4	40 | 17 | 3	38 | 15 | 7	55 | 2 | 3
13	Climate	8 | 15 | 37	21 | 30 | 9	41 | 11 | 8	41 | 15 | 4	24 | 23 | 13	38 | 19 | 3	19 | 31 | 10	32 | 14 | 14	28 | 19 | 13