SEMS not writing cdr to database

Hello,

SEMS has stopped writing records to cdr database. The sems logs show this error in repetition:

[14707/core/AmB2BSession.cpp:704] ERROR: [8-45723920-6030D05F000E01B2-754C5700] media session is missing, can’t update SDP

Yeti-web logs:

[FATAL]: ActiveRecord::StatementInvalid (PG::CheckViolation: ОШИБКА: для строки не найдена секция в отношении “api_requests”#012D ETAIL: Ключ секционирования для неподходящей строки содержит (created_at) = (2021-02-20 08:53:01.685112+00).#012: INSERT INTO “logs”.“api_requests” (“created_at”, “path”, “method”, “status”, “cont roller”, “action”, “page_duration”, “db_duration”, “params”) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9) RETURNING “id”):

config/initializers/instrumentation_notification.rb:44:in block in <main>'#012config/initializers/instrumentation_notification.rb:31:in process_action’

Yeti web node DB logs:

2021-02-20 07:54:01.158 UTC [15711] yeti@yeti ОПЕРАТОР: INSERT INTO “logs”.“api_requests” (“created_at”, “path”, “method”, “status”, “controller”, “action”, “page_duration”, “db_duration”, “params”) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9) RETURNING “id”
2021-02-20 07:54:01.967 UTC [15625] yeti@yeti ОШИБКА: для строки не найдена секция в отношении “api_requests”

Master DB logs:

2021-02-20 09:11:25.740 UTC [8725] cdr@cdr ОПЕРАТОР: SELECT writecdr($1,$2,$3,$4,$5,$6,$7,$8,$9,$10,$11,$12,$13,$14,$15,$16,$17,$18,$19,$20,$21,$22,$23,$24,$25,$26,$27,$28,$29,$30,$31,$32,$33,$34,$35,$36,$37,$38,$39,$40,$41,$42,$43,$44);
2021-02-20 09:11:25.819 UTC [8726] cdr@cdr ОШИБКА: для строки не найдена секция в отношении “streams”
2021-02-20 09:11:25.819 UTC [8726] cdr@cdr ПОДРОБНОСТИ: Ключ секционирования для неподходящей строки содержит (time_start) = (2021-02-20 01:51:54.632954+00).
2021-02-20 09:11:25.819 UTC [8726] cdr@cdr КОНТЕКСТ: SQL-оператор: “INSERT INTO rtp_statistics.streams VALUES( v_rtp_stream_data.)"
функция PL/pgSQL writecdr(boolean,integer,integer,integer,boolean,smallint,character varying,integer,character varying,integer,smallint,character varying,integer,character varying,integer,character varying,character varying,json,boolean,integer,character varying,integer,integer,character varying,integer,character varying,character varying,character varying,character varying,character varying,character varying,integer,boolean,json,json,character varying,character varying,json,smallint,bigint,json,json,boolean,json), строка 253, оператор SQL-оператор
2021-02-20 09:11:25.819 UTC [8726] cdr@cdr ОПЕРАТОР: SELECT writecdr($1,$2,$3,$4,$5,$6,$7,$8,$9,$10,$11,$12,$13,$14,$15,$16,$17,$18,$19,$20,$21,$22,$23,$24,$25,$26,$27,$28,$29,$30,$31,$32,$33,$34,$35,$36,$37,$38,$39,$40,$41,$42,$43,$44);
2021-02-20 09:11:27.183 UTC [8727] postgres@postgres ВАЖНО: пользователь “postgres” не прошёл проверку подлинности (по паролю)
2021-02-20 09:11:27.183 UTC [8727] postgres@postgres ПОДРОБНОСТИ: Пользователь “postgres” не имеет пароля.
Подключение соответствует строке 94 в pg_hba.conf: “host all all ::1/128 md5”
2021-02-20 09:11:27.797 UTC [8728] postgres@postgres ВАЖНО: пользователь “postgres” не прошёл проверку подлинности (по паролю)
2021-02-20 09:11:27.797 UTC [8728] postgres@postgres ПОДРОБНОСТИ: Пользователь “postgres” не имеет пароля.
Подключение соответствует строке 94 в pg_hba.conf: “host all all ::1/128 md5”
2021-02-20 09:11:28.428 UTC [8729] postgres@postgres ВАЖНО: пользователь “postgres” не прошёл проверку подлинности (по паролю)
2021-02-20 09:11:28.428 UTC [8729] postgres@postgres ПОДРОБНОСТИ: Пользователь “postgres” не имеет пароля.
Подключение соответствует строке 94 в pg_hba.conf: “host all all ::1/128 md5”
2021-02-20 09:11:29.253 UTC [8730] postgres@postgres ВАЖНО: пользователь “postgres” не прошёл проверку подлинности (по паролю)
2021-02-20 09:11:29.253 UTC [8730] postgres@postgres ПОДРОБНОСТИ: Пользователь “postgres” не имеет пароля.
Подключение соответствует строке 94 в pg_hba.conf: “host all all ::1/128 md5”
2021-02-20 09:11:30.110 UTC [8732] cdr@cdr ОШИБКА: для строки не найдена секция в отношении “streams”
2021-02-20 09:11:30.110 UTC [8732] cdr@cdr ПОДРОБНОСТИ: Ключ секционирования для неподходящей строки содержит (time_start) = (2021-02-20 00:00:03.626991+00).
2021-02-20 09:11:30.110 UTC [8732] cdr@cdr КОНТЕКСТ: SQL-оператор: "INSERT INTO rtp_statistics.streams VALUES( v_rtp_stream_data.
)”
функция PL/pgSQL writecdr(boolean,integer,integer,integer,boolean,smallint,character varying,integer,character varying,integer,smallint,character varying,integer,character varying,integer,character varying,character varying,json,boolean,integer,character varying,integer,integer,character varying,integer,character varying,character varying,character varying,character varying,character varying,character varying,integer,boolean,json,json,character varying,character varying,json,smallint,bigint,json,json,boolean,json), строка 253, оператор SQL-оператор
2021-02-20 09:11:30.110 UTC [8732] cdr@cdr ОПЕРАТОР: SELECT writecdr($1,$2,$3,$4,$5,$6,$7,$8,$9,$10,$11,$12,$13,$14,$15,$16,$17,$18,$19,$20,$21,$22,$23,$24,$25,$26,$27,$28,$29,$30,$31,$32,$33,$34,$35,$36,$37,$38,$39,$40,$41,$42,$43,$44);
2021-02-20 09:11:30.157 UTC [8733] cdr@cdr ОШИБКА: для строки не найдена секция в отношении “streams”
2021-02-20 09:11:30.157 UTC [8733] cdr@cdr ПОДРОБНОСТИ: Ключ секционирования для неподходящей строки содержит (time_start) = (2021-02-20 01:48:44.511256+00).
2021-02-20 09:11:30.157 UTC [8733] cdr@cdr КОНТЕКСТ: SQL-оператор: “INSERT INTO rtp_statistics.streams VALUES( v_rtp_stream_data.)"
функция PL/pgSQL writecdr(boolean,integer,integer,integer,boolean,smallint,character varying,integer,character varying,integer,smallint,character varying,integer,character varying,integer,character varying,character varying,json,boolean,integer,character varying,integer,integer,character varying,integer,character varying,character varying,character varying,character varying,character varying,character varying,integer,boolean,json,json,character varying,character varying,json,smallint,bigint,json,json,boolean,json), строка 253, оператор SQL-оператор
2021-02-20 09:11:30.157 UTC [8733] cdr@cdr ОПЕРАТОР: SELECT writecdr($1,$2,$3,$4,$5,$6,$7,$8,$9,$10,$11,$12,$13,$14,$15,$16,$17,$18,$19,$20,$21,$22,$23,$24,$25,$26,$27,$28,$29,$30,$31,$32,$33,$34,$35,$36,$37,$38,$39,$40,$41,$42,$43,$44);
2021-02-20 09:11:30.234 UTC [8734] postgres@postgres ВАЖНО: пользователь “postgres” не прошёл проверку подлинности (по паролю)
2021-02-20 09:11:30.234 UTC [8734] postgres@postgres ПОДРОБНОСТИ: Пользователь “postgres” не имеет пароля.
Подключение соответствует строке 94 в pg_hba.conf: “host all all ::1/128 md5”
2021-02-20 09:11:30.845 UTC [8735] cdr@cdr ОШИБКА: для строки не найдена секция в отношении “streams”
2021-02-20 09:11:30.845 UTC [8735] cdr@cdr ПОДРОБНОСТИ: Ключ секционирования для неподходящей строки содержит (time_start) = (2021-02-20 00:00:01.693768+00).
2021-02-20 09:11:30.845 UTC [8735] cdr@cdr КОНТЕКСТ: SQL-оператор: "INSERT INTO rtp_statistics.streams VALUES( v_rtp_stream_data.
)”
функция PL/pgSQL writecdr(boolean,integer,integer,integer,boolean,smallint,character varying,integer,character varying,integer,smallint,character varying,integer,character varying,integer,character varying,character varying,json,boolean,integer,character varying,integer,integer,character varying,integer,character varying,character varying,character varying,character varying,character varying,character varying,integer,boolean,json,json,character varying,character varying,json,smallint,bigint,json,json,boolean,json), строка 253, оператор SQL-оператор
2021-02-20 09:11:30.845 UTC [8735] cdr@cdr ОПЕРАТОР: SELECT writecdr($1,$2,$3,$4,$5,$6,$7,$8,$9,$10,$11,$12,$13,$14,$15,$16,$17,$18,$19,$20,$21,$22,$23,$24,$25,$26,$27,$28,$29,$30,$31,$32,$33,$34,$35,$36,$37,$38,$39,$40,$41,$42,$43,$44);
2021-02-20 09:11:30.892 UTC [8736] cdr@cdr ОШИБКА: для строки не найдена секция в отношении “streams”
2021-02-20 09:11:30.892 UTC [8736] cdr@cdr ПОДРОБНОСТИ: Ключ секционирования для неподходящей строки содержит (time_start) = (2021-02-20 01:47:31.117427+00).
2021-02-20 09:11:30.892 UTC [8736] cdr@cdr КОНТЕКСТ: SQL-оператор: “INSERT INTO rtp_statistics.streams VALUES( v_rtp_stream_data.*)”
функция PL/pgSQL writecdr(boolean,integer,integer,integer,boolean,smallint,character varying,integer,character varying,integer,smallint,character varying,integer,character varying,integer,character varying,character varying,json,boolean,integer,character varying,integer,integer,character varying,integer,character varying,character varying,character varying,character varying,character varying,character varying,integer,boolean,json,json,character varying,character varying,json,smallint,bigint,json,json,boolean,json), строка 253, оператор SQL-оператор

Only recent change in config is the activation of 100rel on origination gateways, and sst on termination gateways. Looks like there is a mismatch of accepted row values?

Database partitions were not created. Check System->Jobs. CdrPartitioning should run every hour.

That was it, thanks dmitry.
I have another question if you don’t mind. I have noticed, however that CDR’s before this fix are lost. Is it correct behavior of the master-slave setup to temporarily store cdrs in the event of a situation like this?

The SEMS logs also print these lines:
ERROR: SQL failover enabled. Trying slave connection
ERROR: SQL cant get profiles. Drop request
This means my replication setup is incorrect? How do I fix this?

CDRs will not lost in properly configured system so something wrong with your setup.

These log lines are not related to CDR writing. These logs related to routing requests, so looks like something happened with routing db too.