Question regarding time formats

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Question regarding time formats

Mikael Ekblom

Hi,

 

We have now implemented a pullaction, that is able to automatically generate usernames and passwords from a mssql database source, that is simulating our HR-system. So far so good.

 

I can also see, that syncope is able to fetch start and end dates automatically from the db-source(date stated as datetime column), but, when we try to sync it the other way, we get an exception:

 

09:40:52.331 ERROR Error parsing value {0} of attribute Slutdatum:93              Method: handleAttribute

java.lang.NullPointerException: null

 

First question:

 

Is it now so, that syncope is able to transform and understand a datetime format quite easily into a correct java.date format when it pulls this information from an external source, but the conversion the other way round is troublesome when pushing/propagating data towards the external resource? Do I need to transform it first before the propagation? Maybe a SimpleDate formatter?

 

In that sense, it is better that it fails the other way around, because we are to fetch the values from the HR and they should not be managed from the syncope environment.  

 

Second question:

 

When a connector has the SEARCH and SYNC operations associated with itself: how is the logic supposed to work if you do have the change log column set and deletes a tuple in the external database? Pull mode is set to incremental. On our end here, even though we delete the tuple in the mssql hr simulation database, the associated account is not deleted within syncope.  Not even with pull mode set to “Full reconciliation” funny enough.

 

What I can see, the pull action handler is not logging anything if the pull mode is not set to full reconciliation either. Syncope and the db do have the associated key values set correctly etc. The change log column is of a datetime format also. Does this has something to do with it?   

 

Regards,

 

       Mikael

 

 

 

 

 

Mikael Ekblom

Systemutvecklare/System developer

Arcada, IT

 

Jan-Magnus Janssons plats 1,

FIN-00560 Helsingfors,

Finland

 

TFn: +358 207 699 467

Mobil: +358 207 699 467

 

Reply | Threaded
Open this post in threaded view
|

Re: Question regarding time formats

ilgrosso
Administrator
Hi Mikael,
see my replies below.

Regards.

On 20/04/2017 10:03, Mikael Ekblom wrote:

Hi,

 

We have now implemented a pullaction, that is able to automatically generate usernames and passwords from a mssql database source, that is simulating our HR-system. So far so good.


Cool.

I can also see, that syncope is able to fetch start and end dates automatically from the db-source(date stated as datetime column), but, when we try to sync it the other way, we get an exception:

 

09:40:52.331 ERROR Error parsing value {0} of attribute Slutdatum:93              Method: handleAttribute

java.lang.NullPointerException: null

 

First question:

 

Is it now so, that syncope is able to transform and understand a datetime format quite easily into a correct java.date format when it pulls this information from an external source, but the conversion the other way round is troublesome when pushing/propagating data towards the external resource? Do I need to transform it first before the propagation? Maybe a SimpleDate formatter? 


When Syncope fetches data from the db-source, we are doing pull.
Consider that such data is arriving in a format according to ConnId specifications; FYI, ConnId supports the following types:

https://github.com/Tirasa/ConnId/blob/connid-1.4.2.0/java/connector-framework/src/main/java/org/identityconnectors/framework/common/FrameworkUtil.java#L191-L212

where, as you can see, there is no Date.

This means that the date values arrive as text, and are parsed by Syncope according to the format specified for the mapped internal schema.

At the moment, however, during propagation the eventual conversion pattern is simply not taken into account:

https://github.com/apache/syncope/blob/2_0_X/core/provisioning-java/src/main/java/org/apache/syncope/core/provisioning/java/MappingManagerImpl.java#L257-L263

There should be room for an improvement here: would you mind to file an issue on JIRA about propagating Date values as formatted strings?

In that sense, it is better that it fails the other way around, because we are to fetch the values from the HR and they should not be managed from the syncope environment.


If such values are not to be used anywhere else in Syncope, then simply define the internal schema as String rather than Date.

Second question:

 

When a connector has the SEARCH and SYNC operations associated with itself: how is the logic supposed to work if you do have the change log column set and deletes a tuple in the external database? Pull mode is set to incremental. On our end here, even though we delete the tuple in the mssql hr simulation database, the associated account is not deleted within syncope.  Not even with pull mode set to “Full reconciliation” funny enough.

 

What I can see, the pull action handler is not logging anything if the pull mode is not set to full reconciliation either. Syncope and the db do have the associated key values set correctly etc. The change log column is of a datetime format also. Does this has something to do with it?


You need to consider that everything related to SYNC mostly depends on how the actual connector implements such operation.
The db-table connector is possibly one of the simplest out there; among its limitation,  it is also not capable to feed Syncope with information about deleted users.

I would suggest to use the more flexible Scripted SQL for the job; you'll need, however, to provide your own Groovy scripts.

You might want to look at

https://github.com/apache/syncope/tree/2_0_X/fit/core-reference/src/test/resources/scriptedsql

as starting point; FYI, look how one of such sample scripts can generate DELETE events to send back to Syncope:

https://github.com/apache/syncope/blob/2_0_X/fit/core-reference/src/test/resources/scriptedsql/SyncScript.groovy#L78

by simply considering an additional 'deleted' column.
-- 
Francesco Chicchiriccò

Tirasa - Open Source Excellence
http://www.tirasa.net/

Member at The Apache Software Foundation
Syncope, Cocoon, Olingo, CXF, OpenJPA, PonyMail
http://home.apache.org/~ilgrosso/
Reply | Threaded
Open this post in threaded view
|

RE: Question regarding time formats

Mikael Ekblom

Hi,

 

Sure,  I can file an issue on Jira. As for now, we do get the dates synchronized in the correct manner from the right source.

 

I was reading about the DB connector and I was sure that it stated that it supported delete as optional:

Authenticate (Optional) Specify the password column configuration property.

·       Create

·       Delete

·       Update

·       Search

·       Schema

·       Test

·       Sync (Optional) Specify the Change Log Column configuration property.

I thought that optional was  a notation for the fact that it could be implemented by having the change log column associated with the resource. I’ll look more into the groovy-scripts. We have had some parsing problems with those, but that might be due to some file encoding issue. Therefore we collected the base master data together from various sources and thought of following the “keep it simple stupid” policy with the db-connector…J  

 

Regards,

 

      Mikael

 

 

From: Francesco Chicchiriccò [mailto:[hidden email]]
Sent: perjantai 21. huhtikuuta 2017 11.52
To: [hidden email]
Subject: Re: Question regarding time formats

 

Hi Mikael,
see my replies below.

Regards.

On 20/04/2017 10:03, Mikael Ekblom wrote:

Hi,

 

We have now implemented a pullaction, that is able to automatically generate usernames and passwords from a mssql database source, that is simulating our HR-system. So far so good.


Cool.


I can also see, that syncope is able to fetch start and end dates automatically from the db-source(date stated as datetime column), but, when we try to sync it the other way, we get an exception:

 

09:40:52.331 ERROR Error parsing value {0} of attribute Slutdatum:93              Method: handleAttribute

java.lang.NullPointerException: null

 

First question:

 

Is it now so, that syncope is able to transform and understand a datetime format quite easily into a correct java.date format when it pulls this information from an external source, but the conversion the other way round is troublesome when pushing/propagating data towards the external resource? Do I need to transform it first before the propagation? Maybe a SimpleDate formatter? 


When Syncope fetches data from the db-source, we are doing pull.
Consider that such data is arriving in a format according to ConnId specifications; FYI, ConnId supports the following types:

https://github.com/Tirasa/ConnId/blob/connid-1.4.2.0/java/connector-framework/src/main/java/org/identityconnectors/framework/common/FrameworkUtil.java#L191-L212

where, as you can see, there is no Date.

This means that the date values arrive as text, and are parsed by Syncope according to the format specified for the mapped internal schema.

At the moment, however, during propagation the eventual conversion pattern is simply not taken into account:

https://github.com/apache/syncope/blob/2_0_X/core/provisioning-java/src/main/java/org/apache/syncope/core/provisioning/java/MappingManagerImpl.java#L257-L263

There should be room for an improvement here: would you mind to file an issue on JIRA about propagating Date values as formatted strings?


In that sense, it is better that it fails the other way around, because we are to fetch the values from the HR and they should not be managed from the syncope environment.


If such values are not to be used anywhere else in Syncope, then simply define the internal schema as String rather than Date.


Second question:

 

When a connector has the SEARCH and SYNC operations associated with itself: how is the logic supposed to work if you do have the change log column set and deletes a tuple in the external database? Pull mode is set to incremental. On our end here, even though we delete the tuple in the mssql hr simulation database, the associated account is not deleted within syncope.  Not even with pull mode set to “Full reconciliation” funny enough.

 

What I can see, the pull action handler is not logging anything if the pull mode is not set to full reconciliation either. Syncope and the db do have the associated key values set correctly etc. The change log column is of a datetime format also. Does this has something to do with it?


You need to consider that everything related to SYNC mostly depends on how the actual connector implements such operation.
The db-table connector is possibly one of the simplest out there; among its limitation,  it is also not capable to feed Syncope with information about deleted users.

I would suggest to use the more flexible Scripted SQL for the job; you'll need, however, to provide your own Groovy scripts.

You might want to look at

https://github.com/apache/syncope/tree/2_0_X/fit/core-reference/src/test/resources/scriptedsql

as starting point; FYI, look how one of such sample scripts can generate DELETE events to send back to Syncope:

https://github.com/apache/syncope/blob/2_0_X/fit/core-reference/src/test/resources/scriptedsql/SyncScript.groovy#L78

by simply considering an additional 'deleted' column.

-- 
Francesco Chicchiriccò
 
Tirasa - Open Source Excellence
http://www.tirasa.net/
 
Member at The Apache Software Foundation
Syncope, Cocoon, Olingo, CXF, OpenJPA, PonyMail
http://home.apache.org/~ilgrosso/
Reply | Threaded
Open this post in threaded view
|

Re: Question regarding time formats

ilgrosso
Administrator
On 21/04/2017 11:32, Mikael Ekblom wrote:

Hi,

 

Sure,  I can file an issue on Jira. As for now, we do get the dates synchronized in the correct manner from the right source.


Don't mind, I went ahead and created

https://issues.apache.org/jira/browse/SYNCOPE-1070

Regards.

I was reading about the DB connector and I was sure that it stated that it supported delete as optional:

Authenticate (Optional) Specify the password column configuration property.

·       Create

·       Delete

·       Update

·       Search

·       Schema

·       Test

·       Sync (Optional) Specify the Change Log Column configuration property.

I thought that optional was  a notation for the fact that it could be implemented by having the change log column associated with the resource. I’ll look more into the groovy-scripts. We have had some parsing problems with those, but that might be due to some file encoding issue. Therefore we collected the base master data together from various sources and thought of following the “keep it simple stupid” policy with the db-connector…J  

 

Regards,

 

      Mikael

 

 

From: Francesco Chicchiriccò [[hidden email]]
Sent: perjantai 21. huhtikuuta 2017 11.52
To: [hidden email]
Subject: Re: Question regarding time formats

 

Hi Mikael,
see my replies below.

Regards.

On 20/04/2017 10:03, Mikael Ekblom wrote:

Hi,

 

We have now implemented a pullaction, that is able to automatically generate usernames and passwords from a mssql database source, that is simulating our HR-system. So far so good.


Cool.


I can also see, that syncope is able to fetch start and end dates automatically from the db-source(date stated as datetime column), but, when we try to sync it the other way, we get an exception:

 

09:40:52.331 ERROR Error parsing value {0} of attribute Slutdatum:93              Method: handleAttribute

java.lang.NullPointerException: null

 

First question:

 

Is it now so, that syncope is able to transform and understand a datetime format quite easily into a correct java.date format when it pulls this information from an external source, but the conversion the other way round is troublesome when pushing/propagating data towards the external resource? Do I need to transform it first before the propagation? Maybe a SimpleDate formatter? 


When Syncope fetches data from the db-source, we are doing pull.
Consider that such data is arriving in a format according to ConnId specifications; FYI, ConnId supports the following types:

https://github.com/Tirasa/ConnId/blob/connid-1.4.2.0/java/connector-framework/src/main/java/org/identityconnectors/framework/common/FrameworkUtil.java#L191-L212

where, as you can see, there is no Date.

This means that the date values arrive as text, and are parsed by Syncope according to the format specified for the mapped internal schema.

At the moment, however, during propagation the eventual conversion pattern is simply not taken into account:

https://github.com/apache/syncope/blob/2_0_X/core/provisioning-java/src/main/java/org/apache/syncope/core/provisioning/java/MappingManagerImpl.java#L257-L263

There should be room for an improvement here: would you mind to file an issue on JIRA about propagating Date values as formatted strings?


In that sense, it is better that it fails the other way around, because we are to fetch the values from the HR and they should not be managed from the syncope environment.


If such values are not to be used anywhere else in Syncope, then simply define the internal schema as String rather than Date.


Second question:

 

When a connector has the SEARCH and SYNC operations associated with itself: how is the logic supposed to work if you do have the change log column set and deletes a tuple in the external database? Pull mode is set to incremental. On our end here, even though we delete the tuple in the mssql hr simulation database, the associated account is not deleted within syncope.  Not even with pull mode set to “Full reconciliation” funny enough.

 

What I can see, the pull action handler is not logging anything if the pull mode is not set to full reconciliation either. Syncope and the db do have the associated key values set correctly etc. The change log column is of a datetime format also. Does this has something to do with it?


You need to consider that everything related to SYNC mostly depends on how the actual connector implements such operation.
The db-table connector is possibly one of the simplest out there; among its limitation,  it is also not capable to feed Syncope with information about deleted users.

I would suggest to use the more flexible Scripted SQL for the job; you'll need, however, to provide your own Groovy scripts.

You might want to look at

https://github.com/apache/syncope/tree/2_0_X/fit/core-reference/src/test/resources/scriptedsql

as starting point; FYI, look how one of such sample scripts can generate DELETE events to send back to Syncope:

https://github.com/apache/syncope/blob/2_0_X/fit/core-reference/src/test/resources/scriptedsql/SyncScript.groovy#L78

by simply considering an additional 'deleted' column.

-- 
Francesco Chicchiriccò
 
Tirasa - Open Source Excellence
http://www.tirasa.net/
 
Member at The Apache Software Foundation
Syncope, Cocoon, Olingo, CXF, OpenJPA, PonyMail
http://home.apache.org/~ilgrosso/



-- 
Francesco Chicchiriccò

Tirasa - Open Source Excellence
http://www.tirasa.net/

Member at The Apache Software Foundation
Syncope, Cocoon, Olingo, CXF, OpenJPA, PonyMail
http://home.apache.org/~ilgrosso/