Kannel Redis - can I force the destination in the key name?

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Kannel Redis - can I force the destination in the key name?

Robert Chen
Hi, I'm trying to set up Kannel with Redis. My concern is that some DLRs would be lost due to timestamp collisions. And my question is whether hard-coding dlr_redis_add to use the destination (at least for my personal installation) is the best approach. 

dlr_redis.c:
Changing:
if (entry->use_dst && entry->destination)
To this:
if (entry->destination) { ... }

Here's my train of thought:

I'm load-balancing Kannel. And according to community recommendations, each instance is using the same smsc-id:
The current Kannel setup is using 1.4.4 with MySQL. I'm in the process of migrating the whole thing to cloud hosting, but using Redis this time.

I'm worried that DLRs could be lost if they come back with the same timestamp. Within dlr_redis.c, it looks like it does one of two things:

1. If the REDIS_PRECHECK flag is set, it uses HSETNX (only creating a key if it does not exist)
2. Otherwise, it uses HMSET (overriding whatever dictionary is tied to that key)

Either way, it looks like one of the DLRs will get lost if there's ever a conflict.

Within dlr_redis.c, it looks like the key can be one of two formats:

1. <table>:<smsc>:<timestamp>
2. <table>:<smsc>:<timestamp>:<destination>

After analyzing a MySQL CSV dump containing 1 million messages, I'm finding rows with identical timestamps. And for Redis, this could spell a key collision. But, I can get all the keys to be unique by including the destination.

I'm trying to figure out how to force Kannel to include the destination. According to the documentation, it seems like this is determined by the SMSC:

“Some SMSCs also append the destination to the DLR keyname resulting DLR keynames in the format <table>:<smsc>:<timestamp>:<destination>.”

It seems as though cimd2 and emi are the SMSC types that append the destination. 

gateway-1.4.4/gw $ grep -iR dlr_add .
./smsc/http/clickatell.c:                dlr_add(conn->id, msgid, msg, 0);
./smsc/http/generic.c:                        dlr_add(conn->id, msgid, msg, 0);
./smsc/http/generic.c:                dlr_add(conn->id, msgid, msg, 0);
./smsc/http/xidris.c:                dlr_add(conn->id, mid, msg, 0);
./smsc/smsc_at.c:                        dlr_add(privdata->conn->id, dlrmsgid, msg, 0);
./smsc/smsc_cgw.c:            dlr_add(conn->id, ts, msg, 0);
./smsc/smsc_cimd2.c:        dlr_add(conn->name, ts, msg, 1);
./smsc/smsc_emi.c:     dlr_add((conn->id ? conn->id : privdata->name), ts, m, 1);
./smsc/smsc_fake.c:        dlr_add(conn->id, tmp, sms, 0);
./smsc/smsc_http.c:            dlr_add(conn->id, mid, msg, 0);
./smsc/smsc_loopback.c:        dlr_add(conn->id, mid, sms, 0);
./smsc/smsc_oisd.c:        dlr_add(conn->name, ts, msg, 0);
./smsc/smsc_smpp.c:                    dlr_add(smpp->conn->id, tmp, msg, 0);
./smsc/smsc_soap.c:        dlr_add(conn->id, octstr_imm(tmpid), msg, 0);
./smsc/smsc_soap_parlayx.c:            dlr_add(conn->id, mid, sms, 0);
./smsc/smsc_soap_parlayx.c:            dlr_add(conn->id, mid, msg, 0);

The problem though is that I'm using smpp. (I verified against my SMSC that the format coming back is indeed <table>:<smsc>:<timestamp>)

So I guess my question is, should I just hard-code my local install to append the destination? Or is there a better approach or some configuration flat I'm just not seeing?

Thanks!
Robert Chen
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Kannel Redis - can I force the destination in the key name?

Donald Jackson-4
Ts is just the old variable name, the remote message id is used for that field, not a time stamp. So as long as your supplier provides a unique message ID you won't have collisions 

On Tuesday, 01 November 2016, Robert Chen <[hidden email]> wrote:
Hi, I'm trying to set up Kannel with Redis. My concern is that some DLRs would be lost due to timestamp collisions. And my question is whether hard-coding dlr_redis_add to use the destination (at least for my personal installation) is the best approach. 

dlr_redis.c:
Changing:
if (entry->use_dst && entry->destination)
To this:
if (entry->destination) { ... }

Here's my train of thought:

I'm load-balancing Kannel. And according to community recommendations, each instance is using the same smsc-id:
The current Kannel setup is using 1.4.4 with MySQL. I'm in the process of migrating the whole thing to cloud hosting, but using Redis this time.

I'm worried that DLRs could be lost if they come back with the same timestamp. Within dlr_redis.c, it looks like it does one of two things:

1. If the REDIS_PRECHECK flag is set, it uses HSETNX (only creating a key if it does not exist)
2. Otherwise, it uses HMSET (overriding whatever dictionary is tied to that key)

Either way, it looks like one of the DLRs will get lost if there's ever a conflict.

Within dlr_redis.c, it looks like the key can be one of two formats:

1. <table>:<smsc>:<timestamp>
2. <table>:<smsc>:<timestamp>:<destination>

After analyzing a MySQL CSV dump containing 1 million messages, I'm finding rows with identical timestamps. And for Redis, this could spell a key collision. But, I can get all the keys to be unique by including the destination.

I'm trying to figure out how to force Kannel to include the destination. According to the documentation, it seems like this is determined by the SMSC:

“Some SMSCs also append the destination to the DLR keyname resulting DLR keynames in the format <table>:<smsc>:<timestamp>:<destination>.”

It seems as though cimd2 and emi are the SMSC types that append the destination. 

gateway-1.4.4/gw $ grep -iR dlr_add .
./smsc/http/clickatell.c:                dlr_add(conn->id, msgid, msg, 0);
./smsc/http/generic.c:                        dlr_add(conn->id, msgid, msg, 0);
./smsc/http/generic.c:                dlr_add(conn->id, msgid, msg, 0);
./smsc/http/xidris.c:                dlr_add(conn->id, mid, msg, 0);
./smsc/smsc_at.c:                        dlr_add(privdata->conn->id, dlrmsgid, msg, 0);
./smsc/smsc_cgw.c:            dlr_add(conn->id, ts, msg, 0);
./smsc/smsc_cimd2.c:        dlr_add(conn->name, ts, msg, 1);
./smsc/smsc_emi.c:     dlr_add((conn->id ? conn->id : privdata->name), ts, m, 1);
./smsc/smsc_fake.c:        dlr_add(conn->id, tmp, sms, 0);
./smsc/smsc_http.c:            dlr_add(conn->id, mid, msg, 0);
./smsc/smsc_loopback.c:        dlr_add(conn->id, mid, sms, 0);
./smsc/smsc_oisd.c:        dlr_add(conn->name, ts, msg, 0);
./smsc/smsc_smpp.c:                    dlr_add(smpp->conn->id, tmp, msg, 0);
./smsc/smsc_soap.c:        dlr_add(conn->id, octstr_imm(tmpid), msg, 0);
./smsc/smsc_soap_parlayx.c:            dlr_add(conn->id, mid, sms, 0);
./smsc/smsc_soap_parlayx.c:            dlr_add(conn->id, mid, msg, 0);

The problem though is that I'm using smpp. (I verified against my SMSC that the format coming back is indeed <table>:<smsc>:<timestamp>)

So I guess my question is, should I just hard-code my local install to append the destination? Or is there a better approach or some configuration flat I'm just not seeing?

Thanks!
Robert Chen


--

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Kannel Redis - can I force the destination in the key name?

Robert Chen
Thanks for the clarification. Turned out that in my case, some of the message IDs would get recycled given enough time and volume. So in order to generate unique redis keys, I still had to include the destination.

In case anyone is interested, I ended up making a local change to gw/smsc/smsc_smpp.c, by setting the use_dst flag to 1 for the dlr_find and dlr_add method calls. Just had to be mindful that the redis key only uses the last 7 characters of the destination.

@@ -1535,7 +1535,7 @@
         dlrmsg = dlr_find(smpp->conn->id,
             tmp, /* smsc message id */
             destination_addr, /* destination */
-            dlrstat, 0);
+            dlrstat, 1);
 
         octstr_destroy(msgid);
     } else
@@ -1854,7 +1854,7 @@
                  * order to get it logged to access-log.
                  */
                 if (DLR_IS_ENABLED_DEVICE(msg->sms.dlr_mask)) {
-                    dlr_add(smpp->conn->id, tmp, msg, 0);
+                    dlr_add(smpp->conn->id, tmp, msg, 1);
                     octstr_destroy(tmp);
                 } else {
                     octstr_destroy(msg->sms.foreign_id);

On Tue, Nov 1, 2016 at 5:33 PM, Donald Jackson <[hidden email]> wrote:
Ts is just the old variable name, the remote message id is used for that field, not a time stamp. So as long as your supplier provides a unique message ID you won't have collisions 


On Tuesday, 01 November 2016, Robert Chen <[hidden email]> wrote:
Hi, I'm trying to set up Kannel with Redis. My concern is that some DLRs would be lost due to timestamp collisions. And my question is whether hard-coding dlr_redis_add to use the destination (at least for my personal installation) is the best approach. 

dlr_redis.c:
Changing:
if (entry->use_dst && entry->destination)
To this:
if (entry->destination) { ... }

Here's my train of thought:

I'm load-balancing Kannel. And according to community recommendations, each instance is using the same smsc-id:
The current Kannel setup is using 1.4.4 with MySQL. I'm in the process of migrating the whole thing to cloud hosting, but using Redis this time.

I'm worried that DLRs could be lost if they come back with the same timestamp. Within dlr_redis.c, it looks like it does one of two things:

1. If the REDIS_PRECHECK flag is set, it uses HSETNX (only creating a key if it does not exist)
2. Otherwise, it uses HMSET (overriding whatever dictionary is tied to that key)

Either way, it looks like one of the DLRs will get lost if there's ever a conflict.

Within dlr_redis.c, it looks like the key can be one of two formats:

1. <table>:<smsc>:<timestamp>
2. <table>:<smsc>:<timestamp>:<destination>

After analyzing a MySQL CSV dump containing 1 million messages, I'm finding rows with identical timestamps. And for Redis, this could spell a key collision. But, I can get all the keys to be unique by including the destination.

I'm trying to figure out how to force Kannel to include the destination. According to the documentation, it seems like this is determined by the SMSC:

“Some SMSCs also append the destination to the DLR keyname resulting DLR keynames in the format <table>:<smsc>:<timestamp>:<destination>.”

It seems as though cimd2 and emi are the SMSC types that append the destination. 

gateway-1.4.4/gw $ grep -iR dlr_add .
./smsc/http/clickatell.c:                dlr_add(conn->id, msgid, msg, 0);
./smsc/http/generic.c:                        dlr_add(conn->id, msgid, msg, 0);
./smsc/http/generic.c:                dlr_add(conn->id, msgid, msg, 0);
./smsc/http/xidris.c:                dlr_add(conn->id, mid, msg, 0);
./smsc/smsc_at.c:                        dlr_add(privdata->conn->id, dlrmsgid, msg, 0);
./smsc/smsc_cgw.c:            dlr_add(conn->id, ts, msg, 0);
./smsc/smsc_cimd2.c:        dlr_add(conn->name, ts, msg, 1);
./smsc/smsc_emi.c:     dlr_add((conn->id ? conn->id : privdata->name), ts, m, 1);
./smsc/smsc_fake.c:        dlr_add(conn->id, tmp, sms, 0);
./smsc/smsc_http.c:            dlr_add(conn->id, mid, msg, 0);
./smsc/smsc_loopback.c:        dlr_add(conn->id, mid, sms, 0);
./smsc/smsc_oisd.c:        dlr_add(conn->name, ts, msg, 0);
./smsc/smsc_smpp.c:                    dlr_add(smpp->conn->id, tmp, msg, 0);
./smsc/smsc_soap.c:        dlr_add(conn->id, octstr_imm(tmpid), msg, 0);
./smsc/smsc_soap_parlayx.c:            dlr_add(conn->id, mid, sms, 0);
./smsc/smsc_soap_parlayx.c:            dlr_add(conn->id, mid, msg, 0);

The problem though is that I'm using smpp. (I verified against my SMSC that the format coming back is indeed <table>:<smsc>:<timestamp>)

So I guess my question is, should I just hard-code my local install to append the destination? Or is there a better approach or some configuration flat I'm just not seeing?

Thanks!
Robert Chen


--


Loading...