[4.1.1] 4.x Cache headaches.

classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

[4.1.1] 4.x Cache headaches.

Nathan Keiter

Just a word of warning for plugin developers moving to 4.x.  The Java APIs in 4.x are highly cache dependent. This is great for server performance in the long term. However in the immediate context of a critical business process, it is problematic.

 

This cache dependence makes deterministic business logic algorithms nearly impossible.

 

For example, if I create a new “Apple” object and save it, I expect to be able to load that “Apple” object, modify it (maybe taking a bite out of it), and save it again. Unfortunately when you go to load your newly saved “Apple”, you get a null pointer exception or some other “not found” error thrown. This is because in 4.x all the APIs hit the cache, and the cache update is so slow that your code is trying to load it before the cache is updated.

 

The most recent example of this we encountered is using the userAPI to create a new user. A little later in the code we try to load the newly created user via the userAPI and get a null pointer exception, even though if we look in the database, the new user has in fact been created!

 

This issue can bite you for anything that is cached. We’ve seen it in content, relationships, and users to name a few. Content does have an isIndexed check, but it only applies to the contentlet itself, not anything related to it, i.e. relationships, permissions etc. So even that check is not reliable in the holistic sense of “Is this thing entirely cached?”

 

This asynchronous “eventually available” philosophy for cache is unreliable for deterministic business processes. Hack-ish solutions include adding a Java sleep to delay your program in the hopes that the cache will catch up by the time you try to load your data, or risking an infinite loop of “is it here yet?” checks. Obviously neither is ideal.

 

It would be nice if there was a “no-cache” API option available so we could reliably load newly created records during critical business logic blocks. Data accuracy and immediate accessibility is of utmost importance in these cases. I can afford to take a slight performance hit to check the DB. And honestly it would probably be less of a performance hit than waiting for the cache to update.

 

We did not have these issues in 3.x.

 

Nathan I. Keiter | Lead Network Applications Programmer | Benefits Advisory Council Member | I.D.E.A Council Member
Gettysburg College | Information Technology | DataSystems
Campus Box 2453 | 300 North Washington Street | Gettysburg, PA 17325
Phone: 717.337.6993

https://www.gettysburg.edu

 

--
http://dotcms.com - Open Source Java Content Management
---
You received this message because you are subscribed to the Google Groups "dotCMS User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/dotcms/c637e1364965480080ea2f147ef92847%40exch13b.ms.gettysburg.edu.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: [4.1.1] 4.x Cache headaches.

wezell1
Nathan:

There is nothing in 4.1x that changed or is async with regards to cache invalidations- the async changes were added in the 4.2.x series and I believe (though am not certain) they can be mostly turned off by adding REINDEX_ON_SAVE_IN_SEPARATE_THREAD=false to your dotmarketing-config.  If you are saving a user and then calling for that user and getting a null, that would be a bug (unless you are in 2 transactions).  If you share your code we would be happy to take a look and either validate it or use it to write a test and fix the case.  In fact, any code where you are seeing this case in 4.1 would be a good candidate for a test.  




On Thu, Jan 4, 2018 at 12:01 PM, Nathan Keiter <[hidden email]> wrote:

Just a word of warning for plugin developers moving to 4.x.  The Java APIs in 4.x are highly cache dependent. This is great for server performance in the long term. However in the immediate context of a critical business process, it is problematic.

 

This cache dependence makes deterministic business logic algorithms nearly impossible.

 

For example, if I create a new “Apple” object and save it, I expect to be able to load that “Apple” object, modify it (maybe taking a bite out of it), and save it again. Unfortunately when you go to load your newly saved “Apple”, you get a null pointer exception or some other “not found” error thrown. This is because in 4.x all the APIs hit the cache, and the cache update is so slow that your code is trying to load it before the cache is updated.

 

The most recent example of this we encountered is using the userAPI to create a new user. A little later in the code we try to load the newly created user via the userAPI and get a null pointer exception, even though if we look in the database, the new user has in fact been created!

 

This issue can bite you for anything that is cached. We’ve seen it in content, relationships, and users to name a few. Content does have an isIndexed check, but it only applies to the contentlet itself, not anything related to it, i.e. relationships, permissions etc. So even that check is not reliable in the holistic sense of “Is this thing entirely cached?”

 

This asynchronous “eventually available” philosophy for cache is unreliable for deterministic business processes. Hack-ish solutions include adding a Java sleep to delay your program in the hopes that the cache will catch up by the time you try to load your data, or risking an infinite loop of “is it here yet?” checks. Obviously neither is ideal.

 

It would be nice if there was a “no-cache” API option available so we could reliably load newly created records during critical business logic blocks. Data accuracy and immediate accessibility is of utmost importance in these cases. I can afford to take a slight performance hit to check the DB. And honestly it would probably be less of a performance hit than waiting for the cache to update.

 

We did not have these issues in 3.x.

 

Nathan I. Keiter | Lead Network Applications Programmer | Benefits Advisory Council Member | I.D.E.A Council Member
Gettysburg College | Information Technology | DataSystems
Campus Box 2453 | 300 North Washington Street | Gettysburg, PA 17325
Phone: <a href="tel:(717)%20337-6993" value="+17173376993" target="_blank">717.337.6993

https://www.gettysburg.edu

 

--
http://dotcms.com - Open Source Java Content Management
---
You received this message because you are subscribed to the Google Groups "dotCMS User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/dotcms/c637e1364965480080ea2f147ef92847%40exch13b.ms.gettysburg.edu.
For more options, visit https://groups.google.com/d/optout.



--



3059 Grand Avenue
Suite 410-B
Miami FL 33133
Main: 
305-900-2001 | Direct: 978.294.9429

   

--
http://dotcms.com - Open Source Java Content Management
---
You received this message because you are subscribed to the Google Groups "dotCMS User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/dotcms/CAPL0VyEObGj3StmJfPD7ig-MNz-MeisMTYs5HyjVXgNoMOEVWw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: [4.1.1] 4.x Cache headaches.

wezell1
I see the user code you submitted to support - I don't think it has anything to do with cache delay.  What is null is not the User but the instance of the UserCache.  Are you trying to use the UserFactoryLiferayImpl directly?    I think it has to do with using the impl factory UserFactoryLiferayImpl directly (statically) rather than calling the method through the UserAPI.  I think you should be calling:

APILocator.getUserAPI().loadUserById(userId)


On Thu, Jan 4, 2018 at 12:16 PM, Will Ezell <[hidden email]> wrote:
Nathan:

There is nothing in 4.1x that changed or is async with regards to cache invalidations- the async changes were added in the 4.2.x series and I believe (though am not certain) they can be mostly turned off by adding REINDEX_ON_SAVE_IN_SEPARATE_THREAD=false to your dotmarketing-config.  If you are saving a user and then calling for that user and getting a null, that would be a bug (unless you are in 2 transactions).  If you share your code we would be happy to take a look and either validate it or use it to write a test and fix the case.  In fact, any code where you are seeing this case in 4.1 would be a good candidate for a test.  




On Thu, Jan 4, 2018 at 12:01 PM, Nathan Keiter <[hidden email]> wrote:

Just a word of warning for plugin developers moving to 4.x.  The Java APIs in 4.x are highly cache dependent. This is great for server performance in the long term. However in the immediate context of a critical business process, it is problematic.

 

This cache dependence makes deterministic business logic algorithms nearly impossible.

 

For example, if I create a new “Apple” object and save it, I expect to be able to load that “Apple” object, modify it (maybe taking a bite out of it), and save it again. Unfortunately when you go to load your newly saved “Apple”, you get a null pointer exception or some other “not found” error thrown. This is because in 4.x all the APIs hit the cache, and the cache update is so slow that your code is trying to load it before the cache is updated.

 

The most recent example of this we encountered is using the userAPI to create a new user. A little later in the code we try to load the newly created user via the userAPI and get a null pointer exception, even though if we look in the database, the new user has in fact been created!

 

This issue can bite you for anything that is cached. We’ve seen it in content, relationships, and users to name a few. Content does have an isIndexed check, but it only applies to the contentlet itself, not anything related to it, i.e. relationships, permissions etc. So even that check is not reliable in the holistic sense of “Is this thing entirely cached?”

 

This asynchronous “eventually available” philosophy for cache is unreliable for deterministic business processes. Hack-ish solutions include adding a Java sleep to delay your program in the hopes that the cache will catch up by the time you try to load your data, or risking an infinite loop of “is it here yet?” checks. Obviously neither is ideal.

 

It would be nice if there was a “no-cache” API option available so we could reliably load newly created records during critical business logic blocks. Data accuracy and immediate accessibility is of utmost importance in these cases. I can afford to take a slight performance hit to check the DB. And honestly it would probably be less of a performance hit than waiting for the cache to update.

 

We did not have these issues in 3.x.

 

Nathan I. Keiter | Lead Network Applications Programmer | Benefits Advisory Council Member | I.D.E.A Council Member
Gettysburg College | Information Technology | DataSystems
Campus Box 2453 | 300 North Washington Street | Gettysburg, PA 17325
Phone: <a href="tel:(717)%20337-6993" value="+17173376993" target="_blank">717.337.6993

https://www.gettysburg.edu

 

--
http://dotcms.com - Open Source Java Content Management
---
You received this message because you are subscribed to the Google Groups "dotCMS User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/dotcms/c637e1364965480080ea2f147ef92847%40exch13b.ms.gettysburg.edu.
For more options, visit https://groups.google.com/d/optout.



--



3059 Grand Avenue
Suite 410-B
Miami FL 33133
Main: 
305-900-2001 | Direct: 978.294.9429

   



--



3059 Grand Avenue
Suite 410-B
Miami FL 33133
Main: 
305-900-2001 | Direct: 978.294.9429

   

--
http://dotcms.com - Open Source Java Content Management
---
You received this message because you are subscribed to the Google Groups "dotCMS User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/dotcms/CAPL0VyGGDVbXoBp8OOyYV__f9O-Qc_dRBBATdUaUmD%2BXH5e2og%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

RE: [4.1.1] 4.x Cache headaches.

Nathan Keiter

No, I am getting it via the APILocator higher up in the code.

 

I’ll continue with this on the support ticket.

 

However, like I said, typically this goes away by adding a java sleep for enough milliseconds…

 

Nathan I. Keiter | Lead Network Applications Programmer | Benefits Advisory Council Member | I.D.E.A Council Member
Gettysburg College | Information Technology | DataSystems
Campus Box 2453 | 300 North Washington Street | Gettysburg, PA 17325
Phone: 717.337.6993

https://www.gettysburg.edu

 

From: [hidden email] [mailto:[hidden email]] On Behalf Of Will Ezell
Sent: Thursday, January 04, 2018 12:26 PM
To: [hidden email]
Subject: Re: [dotcms] [4.1.1] 4.x Cache headaches.

 

I see the user code you submitted to support - I don't think it has anything to do with cache delay.  What is null is not the User but the instance of the UserCache.  Are you trying to use the UserFactoryLiferayImpl directly?    I think it has to do with using the impl factory UserFactoryLiferayImpl directly (statically) rather than calling the method through the UserAPI.  I think you should be calling:


APILocator.getUserAPI().loadUserById(userId)

 

On Thu, Jan 4, 2018 at 12:16 PM, Will Ezell <[hidden email]> wrote:

Nathan:

 

There is nothing in 4.1x that changed or is async with regards to cache invalidations- the async changes were added in the 4.2.x series and I believe (though am not certain) they can be mostly turned off by adding REINDEX_ON_SAVE_IN_SEPARATE_THREAD=false to your dotmarketing-config.  If you are saving a user and then calling for that user and getting a null, that would be a bug (unless you are in 2 transactions).  If you share your code we would be happy to take a look and either validate it or use it to write a test and fix the case.  In fact, any code where you are seeing this case in 4.1 would be a good candidate for a test.  

 

 

 

 

On Thu, Jan 4, 2018 at 12:01 PM, Nathan Keiter <[hidden email]> wrote:

Just a word of warning for plugin developers moving to 4.x.  The Java APIs in 4.x are highly cache dependent. This is great for server performance in the long term. However in the immediate context of a critical business process, it is problematic.

 

This cache dependence makes deterministic business logic algorithms nearly impossible.

 

For example, if I create a new “Apple” object and save it, I expect to be able to load that “Apple” object, modify it (maybe taking a bite out of it), and save it again. Unfortunately when you go to load your newly saved “Apple”, you get a null pointer exception or some other “not found” error thrown. This is because in 4.x all the APIs hit the cache, and the cache update is so slow that your code is trying to load it before the cache is updated.

 

The most recent example of this we encountered is using the userAPI to create a new user. A little later in the code we try to load the newly created user via the userAPI and get a null pointer exception, even though if we look in the database, the new user has in fact been created!

 

This issue can bite you for anything that is cached. We’ve seen it in content, relationships, and users to name a few. Content does have an isIndexed check, but it only applies to the contentlet itself, not anything related to it, i.e. relationships, permissions etc. So even that check is not reliable in the holistic sense of “Is this thing entirely cached?”

 

This asynchronous “eventually available” philosophy for cache is unreliable for deterministic business processes. Hack-ish solutions include adding a Java sleep to delay your program in the hopes that the cache will catch up by the time you try to load your data, or risking an infinite loop of “is it here yet?” checks. Obviously neither is ideal.

 

It would be nice if there was a “no-cache” API option available so we could reliably load newly created records during critical business logic blocks. Data accuracy and immediate accessibility is of utmost importance in these cases. I can afford to take a slight performance hit to check the DB. And honestly it would probably be less of a performance hit than waiting for the cache to update.

 

We did not have these issues in 3.x.

 

Nathan I. Keiter | Lead Network Applications Programmer | Benefits Advisory Council Member | I.D.E.A Council Member
Gettysburg College | Information Technology | DataSystems
Campus Box <a href="https://maps.google.com/?q=2453&#43;%7C&#43;300&#43;North&#43;Washington&#43;Street&#43;%7C&#43;Gettysburg,&#43;PA&#43;17325&amp;entry=gmail&amp;source=g" target="_blank"> 2453 | 300 North Washington Street | Gettysburg, PA 17325
Phone: <a href="tel:(717)%20337-6993" target="_blank"> 717.337.6993

https://www.gettysburg.edu

 

--
http://dotcms.com - Open Source Java Content Management
---
You received this message because you are subscribed to the Google Groups "dotCMS User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/dotcms/c637e1364965480080ea2f147ef92847%40exch13b.ms.gettysburg.edu.
For more options, visit https://groups.google.com/d/optout.



 

--

 

 

3059 Grand Avenue

Suite 410-B

Miami FL 33133

Main: 305-900-2001 | Direct: 978.294.9429

 

   



 

--

 

 

3059 Grand Avenue

Suite 410-B

Miami FL 33133

Main: 305-900-2001 | Direct: 978.294.9429

 

   

--
http://dotcms.com - Open Source Java Content Management
---
You received this message because you are subscribed to the Google Groups "dotCMS User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/dotcms/CAPL0VyGGDVbXoBp8OOyYV__f9O-Qc_dRBBATdUaUmD%2BXH5e2og%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
http://dotcms.com - Open Source Java Content Management
---
You received this message because you are subscribed to the Google Groups "dotCMS User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/dotcms/e889adbfb07047de80634b693ef26a93%40exch13b.ms.gettysburg.edu.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

RE: [4.1.1] 4.x Cache headaches.

Nathan Keiter

I’ve been dealing with and researching this issue for several weeks now on two different 4.1.1 instances we have. I am very sure it’s a cache delay issue.

 

This does look promising: REINDEX_ON_SAVE_IN_SEPARATE_THREAD=false

 

Nathan I. Keiter | Lead Network Applications Programmer | Benefits Advisory Council Member | I.D.E.A Council Member
Gettysburg College | Information Technology | DataSystems
Campus Box 2453 | 300 North Washington Street | Gettysburg, PA 17325
Phone: 717.337.6993

https://www.gettysburg.edu

 

From: [hidden email] [mailto:[hidden email]] On Behalf Of Nathan Keiter
Sent: Thursday, January 04, 2018 12:30 PM
To: [hidden email]
Subject: RE: [dotcms] [4.1.1] 4.x Cache headaches.

 

No, I am getting it via the APILocator higher up in the code.

 

I’ll continue with this on the support ticket.

 

However, like I said, typically this goes away by adding a java sleep for enough milliseconds…

 

Nathan I. Keiter | Lead Network Applications Programmer | Benefits Advisory Council Member | I.D.E.A Council Member
Gettysburg College | Information Technology | DataSystems
Campus Box 2453 | 300 North Washington Street | Gettysburg, PA 17325
Phone: 717.337.6993

https://www.gettysburg.edu

 

From: [hidden email] [[hidden email]] On Behalf Of Will Ezell
Sent: Thursday, January 04, 2018 12:26 PM
To: [hidden email]
Subject: Re: [dotcms] [4.1.1] 4.x Cache headaches.

 

I see the user code you submitted to support - I don't think it has anything to do with cache delay.  What is null is not the User but the instance of the UserCache.  Are you trying to use the UserFactoryLiferayImpl directly?    I think it has to do with using the impl factory UserFactoryLiferayImpl directly (statically) rather than calling the method through the UserAPI.  I think you should be calling:

APILocator.getUserAPI().loadUserById(userId)

 

On Thu, Jan 4, 2018 at 12:16 PM, Will Ezell <[hidden email]> wrote:

Nathan:

 

There is nothing in 4.1x that changed or is async with regards to cache invalidations- the async changes were added in the 4.2.x series and I believe (though am not certain) they can be mostly turned off by adding REINDEX_ON_SAVE_IN_SEPARATE_THREAD=false to your dotmarketing-config.  If you are saving a user and then calling for that user and getting a null, that would be a bug (unless you are in 2 transactions).  If you share your code we would be happy to take a look and either validate it or use it to write a test and fix the case.  In fact, any code where you are seeing this case in 4.1 would be a good candidate for a test.  

 

 

 

 

On Thu, Jan 4, 2018 at 12:01 PM, Nathan Keiter <[hidden email]> wrote:

Just a word of warning for plugin developers moving to 4.x.  The Java APIs in 4.x are highly cache dependent. This is great for server performance in the long term. However in the immediate context of a critical business process, it is problematic.

 

This cache dependence makes deterministic business logic algorithms nearly impossible.

 

For example, if I create a new “Apple” object and save it, I expect to be able to load that “Apple” object, modify it (maybe taking a bite out of it), and save it again. Unfortunately when you go to load your newly saved “Apple”, you get a null pointer exception or some other “not found” error thrown. This is because in 4.x all the APIs hit the cache, and the cache update is so slow that your code is trying to load it before the cache is updated.

 

The most recent example of this we encountered is using the userAPI to create a new user. A little later in the code we try to load the newly created user via the userAPI and get a null pointer exception, even though if we look in the database, the new user has in fact been created!

 

This issue can bite you for anything that is cached. We’ve seen it in content, relationships, and users to name a few. Content does have an isIndexed check, but it only applies to the contentlet itself, not anything related to it, i.e. relationships, permissions etc. So even that check is not reliable in the holistic sense of “Is this thing entirely cached?”

 

This asynchronous “eventually available” philosophy for cache is unreliable for deterministic business processes. Hack-ish solutions include adding a Java sleep to delay your program in the hopes that the cache will catch up by the time you try to load your data, or risking an infinite loop of “is it here yet?” checks. Obviously neither is ideal.

 

It would be nice if there was a “no-cache” API option available so we could reliably load newly created records during critical business logic blocks. Data accuracy and immediate accessibility is of utmost importance in these cases. I can afford to take a slight performance hit to check the DB. And honestly it would probably be less of a performance hit than waiting for the cache to update.

 

We did not have these issues in 3.x.

 

Nathan I. Keiter | Lead Network Applications Programmer | Benefits Advisory Council Member | I.D.E.A Council Member
Gettysburg College | Information Technology | DataSystems
Campus Box <a href="https://maps.google.com/?q=2453&#43;%7C&#43;300&#43;North&#43;Washington&#43;Street&#43;%7C&#43;Gettysburg,&#43;PA&#43;17325&amp;entry=gmail&amp;source=g" target="_blank"> 2453 | 300 North Washington Street | Gettysburg, PA 17325
Phone: <a href="tel:(717)%20337-6993" target="_blank"> 717.337.6993

https://www.gettysburg.edu

 

--
http://dotcms.com - Open Source Java Content Management
---
You received this message because you are subscribed to the Google Groups "dotCMS User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/dotcms/c637e1364965480080ea2f147ef92847%40exch13b.ms.gettysburg.edu.
For more options, visit https://groups.google.com/d/optout.



 

--

 

 

3059 Grand Avenue

Suite 410-B

Miami FL 33133

Main: 305-900-2001 | Direct: 978.294.9429

 

   



 

--

 

 

3059 Grand Avenue

Suite 410-B

Miami FL 33133

Main: 305-900-2001 | Direct: 978.294.9429

 

   

--
http://dotcms.com - Open Source Java Content Management
---
You received this message because you are subscribed to the Google Groups "dotCMS User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/dotcms/CAPL0VyGGDVbXoBp8OOyYV__f9O-Qc_dRBBATdUaUmD%2BXH5e2og%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
http://dotcms.com - Open Source Java Content Management
---
You received this message because you are subscribed to the Google Groups "dotCMS User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/dotcms/e889adbfb07047de80634b693ef26a93%40exch13b.ms.gettysburg.edu.
For more options, visit https://groups.google.com/d/optout.

--
http://dotcms.com - Open Source Java Content Management
---
You received this message because you are subscribed to the Google Groups "dotCMS User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/dotcms/5f4dcd1ae1db45c9ab378ec7189d7b7a%40exch13b.ms.gettysburg.edu.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: [4.1.1] 4.x Cache headaches.

wezell1
Please give us a code snippet so we can try to reproduce and write a unit test against.  If it is a code issue, we will get it fixed.

On Thu, Jan 4, 2018 at 12:51 PM, Nathan Keiter <[hidden email]> wrote:

I’ve been dealing with and researching this issue for several weeks now on two different 4.1.1 instances we have. I am very sure it’s a cache delay issue.

 

This does look promising: REINDEX_ON_SAVE_IN_SEPARATE_THREAD=false

 

Nathan I. Keiter | Lead Network Applications Programmer | Benefits Advisory Council Member | I.D.E.A Council Member
Gettysburg College | Information Technology | DataSystems
Campus Box 2453 | 300 North Washington Street | Gettysburg, PA 17325
Phone: <a href="tel:(717)%20337-6993" value="+17173376993" target="_blank">717.337.6993

https://www.gettysburg.edu

 

From: [hidden email] [mailto:[hidden email]] On Behalf Of Nathan Keiter
Sent: Thursday, January 04, 2018 12:30 PM
To: [hidden email]
Subject: RE: [dotcms] [4.1.1] 4.x Cache headaches.

 

No, I am getting it via the APILocator higher up in the code.

 

I’ll continue with this on the support ticket.

 

However, like I said, typically this goes away by adding a java sleep for enough milliseconds…

 

Nathan I. Keiter | Lead Network Applications Programmer | Benefits Advisory Council Member | I.D.E.A Council Member
Gettysburg College | Information Technology | DataSystems
Campus Box 2453 | 300 North Washington Street | Gettysburg, PA 17325
Phone: <a href="tel:(717)%20337-6993" value="+17173376993" target="_blank">717.337.6993

https://www.gettysburg.edu

 

From: [hidden email] [[hidden email]] On Behalf Of Will Ezell
Sent: Thursday, January 04, 2018 12:26 PM
To: [hidden email]
Subject: Re: [dotcms] [4.1.1] 4.x Cache headaches.

 

I see the user code you submitted to support - I don't think it has anything to do with cache delay.  What is null is not the User but the instance of the UserCache.  Are you trying to use the UserFactoryLiferayImpl directly?    I think it has to do with using the impl factory UserFactoryLiferayImpl directly (statically) rather than calling the method through the UserAPI.  I think you should be calling:

APILocator.getUserAPI().loadUserById(userId)

 

On Thu, Jan 4, 2018 at 12:16 PM, Will Ezell <[hidden email]> wrote:

Nathan:

 

There is nothing in 4.1x that changed or is async with regards to cache invalidations- the async changes were added in the 4.2.x series and I believe (though am not certain) they can be mostly turned off by adding REINDEX_ON_SAVE_IN_SEPARATE_THREAD=false to your dotmarketing-config.  If you are saving a user and then calling for that user and getting a null, that would be a bug (unless you are in 2 transactions).  If you share your code we would be happy to take a look and either validate it or use it to write a test and fix the case.  In fact, any code where you are seeing this case in 4.1 would be a good candidate for a test.  

 

 

 

 

On Thu, Jan 4, 2018 at 12:01 PM, Nathan Keiter <[hidden email]> wrote:

Just a word of warning for plugin developers moving to 4.x.  The Java APIs in 4.x are highly cache dependent. This is great for server performance in the long term. However in the immediate context of a critical business process, it is problematic.

 

This cache dependence makes deterministic business logic algorithms nearly impossible.

 

For example, if I create a new “Apple” object and save it, I expect to be able to load that “Apple” object, modify it (maybe taking a bite out of it), and save it again. Unfortunately when you go to load your newly saved “Apple”, you get a null pointer exception or some other “not found” error thrown. This is because in 4.x all the APIs hit the cache, and the cache update is so slow that your code is trying to load it before the cache is updated.

 

The most recent example of this we encountered is using the userAPI to create a new user. A little later in the code we try to load the newly created user via the userAPI and get a null pointer exception, even though if we look in the database, the new user has in fact been created!

 

This issue can bite you for anything that is cached. We’ve seen it in content, relationships, and users to name a few. Content does have an isIndexed check, but it only applies to the contentlet itself, not anything related to it, i.e. relationships, permissions etc. So even that check is not reliable in the holistic sense of “Is this thing entirely cached?”

 

This asynchronous “eventually available” philosophy for cache is unreliable for deterministic business processes. Hack-ish solutions include adding a Java sleep to delay your program in the hopes that the cache will catch up by the time you try to load your data, or risking an infinite loop of “is it here yet?” checks. Obviously neither is ideal.

 

It would be nice if there was a “no-cache” API option available so we could reliably load newly created records during critical business logic blocks. Data accuracy and immediate accessibility is of utmost importance in these cases. I can afford to take a slight performance hit to check the DB. And honestly it would probably be less of a performance hit than waiting for the cache to update.

 

We did not have these issues in 3.x.

 

Nathan I. Keiter | Lead Network Applications Programmer | Benefits Advisory Council Member | I.D.E.A Council Member
Gettysburg College | Information Technology | DataSystems
Campus Box 2453 | 300 North Washington Street | Gettysburg, PA 17325
Phone: <a href="tel:(717)%20337-6993" target="_blank"> 717.337.6993

https://www.gettysburg.edu

 

--
http://dotcms.com - Open Source Java Content Management
---
You received this message because you are subscribed to the Google Groups "dotCMS User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/dotcms/c637e1364965480080ea2f147ef92847%40exch13b.ms.gettysburg.edu.
For more options, visit https://groups.google.com/d/optout.



 

--

 

 

Miami FL 33133

Main: <a href="tel:(305)%20900-2001" value="+13059002001" target="_blank">305-900-2001 | Direct: <a href="tel:(978)%20294-9429" value="+19782949429" target="_blank">978.294.9429

 

   



 

--

 

 

Miami FL 33133

Main: <a href="tel:(305)%20900-2001" value="+13059002001" target="_blank">305-900-2001 | Direct: <a href="tel:(978)%20294-9429" value="+19782949429" target="_blank">978.294.9429

 

   

--
http://dotcms.com - Open Source Java Content Management
---
You received this message because you are subscribed to the Google Groups "dotCMS User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/dotcms/CAPL0VyGGDVbXoBp8OOyYV__f9O-Qc_dRBBATdUaUmD%2BXH5e2og%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
http://dotcms.com - Open Source Java Content Management
---
You received this message because you are subscribed to the Google Groups "dotCMS User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/dotcms/e889adbfb07047de80634b693ef26a93%40exch13b.ms.gettysburg.edu.
For more options, visit https://groups.google.com/d/optout.

--
http://dotcms.com - Open Source Java Content Management
---
You received this message because you are subscribed to the Google Groups "dotCMS User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/dotcms/5f4dcd1ae1db45c9ab378ec7189d7b7a%40exch13b.ms.gettysburg.edu.

For more options, visit https://groups.google.com/d/optout.



--



3059 Grand Avenue
Suite 410-B
Miami FL 33133
Main: 
305-900-2001 | Direct: 978.294.9429

   

--
http://dotcms.com - Open Source Java Content Management
---
You received this message because you are subscribed to the Google Groups "dotCMS User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/dotcms/CAPL0VyFVpP%2BFfHvKKOGCv3C%2Br45xaBrUPu8bhKscUovvr_xZ%2BA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

RE: [4.1.1] 4.x Cache headaches.

Nathan Keiter

Ok, I added a snippet to the ticket. The fact remains that this code worked 100% in 3.5.1, so it seems to point to an issue with 4.1.1. The delta is there.

 

Nathan I. Keiter | Lead Network Applications Programmer | Benefits Advisory Council Member | I.D.E.A Council Member
Gettysburg College | Information Technology | DataSystems
Campus Box 2453 | 300 North Washington Street | Gettysburg, PA 17325
Phone: 717.337.6993

https://www.gettysburg.edu

 

From: [hidden email] [mailto:[hidden email]] On Behalf Of Will Ezell
Sent: Thursday, January 04, 2018 1:28 PM
To: [hidden email]
Subject: Re: [dotcms] [4.1.1] 4.x Cache headaches.

 

Please give us a code snippet so we can try to reproduce and write a unit test against.  If it is a code issue, we will get it fixed.

 

On Thu, Jan 4, 2018 at 12:51 PM, Nathan Keiter <[hidden email]> wrote:

I’ve been dealing with and researching this issue for several weeks now on two different 4.1.1 instances we have. I am very sure it’s a cache delay issue.

 

This does look promising: REINDEX_ON_SAVE_IN_SEPARATE_THREAD=false

 

Nathan I. Keiter | Lead Network Applications Programmer | Benefits Advisory Council Member | I.D.E.A Council Member
Gettysburg College | Information Technology | DataSystems
Campus Box <a href="https://maps.google.com/?q=2453&#43;%7C&#43;300&#43;North&#43;Washington&#43;Street&#43;%7C&#43;Gettysburg,&#43;PA&#43;17325&amp;entry=gmail&amp;source=g"> 2453 | 300 North Washington Street | Gettysburg, PA 17325
Phone: <a href="tel:(717)%20337-6993" target="_blank"> 717.337.6993

https://www.gettysburg.edu

 

From: [hidden email] [mailto:[hidden email]] On Behalf Of Nathan Keiter
Sent: Thursday, January 04, 2018 12:30 PM
To: [hidden email]
Subject: RE: [dotcms] [4.1.1] 4.x Cache headaches.

 

No, I am getting it via the APILocator higher up in the code.

 

I’ll continue with this on the support ticket.

 

However, like I said, typically this goes away by adding a java sleep for enough milliseconds…

 

Nathan I. Keiter | Lead Network Applications Programmer | Benefits Advisory Council Member | I.D.E.A Council Member
Gettysburg College | Information Technology | DataSystems
Campus Box <a href="https://maps.google.com/?q=2453&#43;%7C&#43;300&#43;North&#43;Washington&#43;Street&#43;%7C&#43;Gettysburg,&#43;PA&#43;17325&amp;entry=gmail&amp;source=g"> 2453 | 300 North Washington Street | Gettysburg, PA 17325
Phone: <a href="tel:(717)%20337-6993" target="_blank"> 717.337.6993

https://www.gettysburg.edu

 

From: [hidden email] [[hidden email]] On Behalf Of Will Ezell
Sent: Thursday, January 04, 2018 12:26 PM
To: [hidden email]
Subject: Re: [dotcms] [4.1.1] 4.x Cache headaches.

 

I see the user code you submitted to support - I don't think it has anything to do with cache delay.  What is null is not the User but the instance of the UserCache.  Are you trying to use the UserFactoryLiferayImpl directly?    I think it has to do with using the impl factory UserFactoryLiferayImpl directly (statically) rather than calling the method through the UserAPI.  I think you should be calling:

APILocator.getUserAPI().loadUserById(userId)

 

On Thu, Jan 4, 2018 at 12:16 PM, Will Ezell <[hidden email]> wrote:

Nathan:

 

There is nothing in 4.1x that changed or is async with regards to cache invalidations- the async changes were added in the 4.2.x series and I believe (though am not certain) they can be mostly turned off by adding REINDEX_ON_SAVE_IN_SEPARATE_THREAD=false to your dotmarketing-config.  If you are saving a user and then calling for that user and getting a null, that would be a bug (unless you are in 2 transactions).  If you share your code we would be happy to take a look and either validate it or use it to write a test and fix the case.  In fact, any code where you are seeing this case in 4.1 would be a good candidate for a test.  

 

 

 

 

On Thu, Jan 4, 2018 at 12:01 PM, Nathan Keiter <[hidden email]> wrote:

Just a word of warning for plugin developers moving to 4.x.  The Java APIs in 4.x are highly cache dependent. This is great for server performance in the long term. However in the immediate context of a critical business process, it is problematic.

 

This cache dependence makes deterministic business logic algorithms nearly impossible.

 

For example, if I create a new “Apple” object and save it, I expect to be able to load that “Apple” object, modify it (maybe taking a bite out of it), and save it again. Unfortunately when you go to load your newly saved “Apple”, you get a null pointer exception or some other “not found” error thrown. This is because in 4.x all the APIs hit the cache, and the cache update is so slow that your code is trying to load it before the cache is updated.

 

The most recent example of this we encountered is using the userAPI to create a new user. A little later in the code we try to load the newly created user via the userAPI and get a null pointer exception, even though if we look in the database, the new user has in fact been created!

 

This issue can bite you for anything that is cached. We’ve seen it in content, relationships, and users to name a few. Content does have an isIndexed check, but it only applies to the contentlet itself, not anything related to it, i.e. relationships, permissions etc. So even that check is not reliable in the holistic sense of “Is this thing entirely cached?”

 

This asynchronous “eventually available” philosophy for cache is unreliable for deterministic business processes. Hack-ish solutions include adding a Java sleep to delay your program in the hopes that the cache will catch up by the time you try to load your data, or risking an infinite loop of “is it here yet?” checks. Obviously neither is ideal.

 

It would be nice if there was a “no-cache” API option available so we could reliably load newly created records during critical business logic blocks. Data accuracy and immediate accessibility is of utmost importance in these cases. I can afford to take a slight performance hit to check the DB. And honestly it would probably be less of a performance hit than waiting for the cache to update.

 

We did not have these issues in 3.x.

 

Nathan I. Keiter | Lead Network Applications Programmer | Benefits Advisory Council Member | I.D.E.A Council Member
Gettysburg College | Information Technology | DataSystems
Campus Box <a href="https://maps.google.com/?q=2453&#43;%7C&#43;300&#43;North&#43;Washington&#43;Street&#43;%7C&#43;Gettysburg,&#43;PA&#43;17325&amp;entry=gmail&amp;source=g" target="_blank"> 2453 | 300 North Washington Street | Gettysburg, PA 17325
Phone: <a href="tel:(717)%20337-6993" target="_blank"> 717.337.6993

https://www.gettysburg.edu

 

--
http://dotcms.com - Open Source Java Content Management
---
You received this message because you are subscribed to the Google Groups "dotCMS User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/dotcms/c637e1364965480080ea2f147ef92847%40exch13b.ms.gettysburg.edu.
For more options, visit https://groups.google.com/d/optout.



 

--

 

 

<a href="https://maps.google.com/?q=3059&#43;Grand&#43;Avenue%0D&#43;%0D&#43;%0D&#43;Suite&#43;410&amp;entry=gmail&amp;source=g">3059 Grand Avenue

<a href="https://maps.google.com/?q=3059&#43;Grand&#43;Avenue%0D&#43;%0D&#43;%0D&#43;Suite&#43;410&amp;entry=gmail&amp;source=g">Suite 410-B

Miami FL 33133

Main: <a href="tel:(305)%20900-2001" target="_blank">305-900-2001 | Direct: <a href="tel:(978)%20294-9429" target="_blank">978.294.9429

 

   



 

--

 

 

<a href="https://maps.google.com/?q=3059&#43;Grand&#43;Avenue%0D&#43;%0D&#43;%0D&#43;Suite&#43;410&amp;entry=gmail&amp;source=g">3059 Grand Avenue

<a href="https://maps.google.com/?q=3059&#43;Grand&#43;Avenue%0D&#43;%0D&#43;%0D&#43;Suite&#43;410&amp;entry=gmail&amp;source=g">Suite 410-B

Miami FL 33133

Main: <a href="tel:(305)%20900-2001" target="_blank">305-900-2001 | Direct: <a href="tel:(978)%20294-9429" target="_blank">978.294.9429

 

   

--
http://dotcms.com - Open Source Java Content Management
---
You received this message because you are subscribed to the Google Groups "dotCMS User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/dotcms/CAPL0VyGGDVbXoBp8OOyYV__f9O-Qc_dRBBATdUaUmD%2BXH5e2og%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
http://dotcms.com - Open Source Java Content Management
---
You received this message because you are subscribed to the Google Groups "dotCMS User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/dotcms/e889adbfb07047de80634b693ef26a93%40exch13b.ms.gettysburg.edu.
For more options, visit https://groups.google.com/d/optout.

--
http://dotcms.com - Open Source Java Content Management
---
You received this message because you are subscribed to the Google Groups "dotCMS User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].

To view this discussion on the web visit https://groups.google.com/d/msgid/dotcms/5f4dcd1ae1db45c9ab378ec7189d7b7a%40exch13b.ms.gettysburg.edu.


For more options, visit https://groups.google.com/d/optout.



 

--

 

 

3059 Grand Avenue

Suite 410-B

Miami FL 33133

Main: 305-900-2001 | Direct: 978.294.9429

 

   

--
http://dotcms.com - Open Source Java Content Management
---
You received this message because you are subscribed to the Google Groups "dotCMS User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/dotcms/CAPL0VyFVpP%2BFfHvKKOGCv3C%2Br45xaBrUPu8bhKscUovvr_xZ%2BA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
http://dotcms.com - Open Source Java Content Management
---
You received this message because you are subscribed to the Google Groups "dotCMS User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/dotcms/ca3357b3fe34460399c3807d69fd0fa5%40exch13b.ms.gettysburg.edu.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: [4.1.1] 4.x Cache headaches.

Nathan Keiter
Here's the hack I have to use to get related content after a fresh content save due to the cache delay:


[cid:d0199436-0727-4550-bc26-207a107be322]


Nathan I. Keiter | Lead Network Applications Programmer | Benefits Advisory Council Member | I.D.E.A Council Member
Gettysburg College | Information Technology | DataSystems
Campus Box 2453 | 300 North Washington Street | Gettysburg, PA 17325
Phone: 717.337.6993
https://www.gettysburg.edu
________________________________
From: [hidden email] <[hidden email]> on behalf of Nathan Keiter <[hidden email]>
Sent: Thursday, January 4, 2018 2:22 PM
To: [hidden email]
Subject: RE: [dotcms] [4.1.1] 4.x Cache headaches.

Ok, I added a snippet to the ticket. The fact remains that this code worked 100% in 3.5.1, so it seems to point to an issue with 4.1.1. The delta is there.

Nathan I. Keiter | Lead Network Applications Programmer | Benefits Advisory Council Member | I.D.E.A Council Member
Gettysburg College | Information Technology | DataSystems
Campus Box 2453 | 300 North Washington Street | Gettysburg, PA 17325
Phone: 717.337.6993
https://www.gettysburg.edu<https://www.gettysburg.edu/>

From: [hidden email] [mailto:[hidden email]] On Behalf Of Will Ezell
Sent: Thursday, January 04, 2018 1:28 PM
To: [hidden email]
Subject: Re: [dotcms] [4.1.1] 4.x Cache headaches.

Please give us a code snippet so we can try to reproduce and write a unit test against.  If it is a code issue, we will get it fixed.

On Thu, Jan 4, 2018 at 12:51 PM, Nathan Keiter <[hidden email]<mailto:[hidden email]>> wrote:
I’ve been dealing with and researching this issue for several weeks now on two different 4.1.1 instances we have. I am very sure it’s a cache delay issue.

This does look promising: REINDEX_ON_SAVE_IN_SEPARATE_THREAD=false

Nathan I. Keiter | Lead Network Applications Programmer | Benefits Advisory Council Member | I.D.E.A Council Member
Gettysburg College | Information Technology | DataSystems
Campus Box 2453 | 300 North Washington Street | Gettysburg, PA 17325<https://maps.google.com/?q=2453+%7C+300+North+Washington+Street+%7C+Gettysburg,+PA+17325&entry=gmail&source=g>
Phone: 717.337.6993<tel:(717)%20337-6993>
https://www.gettysburg.edu<https://www.gettysburg.edu/>

From: [hidden email]<mailto:[hidden email]> [mailto:[hidden email]<mailto:[hidden email]>] On Behalf Of Nathan Keiter
Sent: Thursday, January 04, 2018 12:30 PM
To: [hidden email]<mailto:[hidden email]>
Subject: RE: [dotcms] [4.1.1] 4.x Cache headaches.

No, I am getting it via the APILocator higher up in the code.

I’ll continue with this on the support ticket.

However, like I said, typically this goes away by adding a java sleep for enough milliseconds…

Nathan I. Keiter | Lead Network Applications Programmer | Benefits Advisory Council Member | I.D.E.A Council Member
Gettysburg College | Information Technology | DataSystems
Campus Box 2453 | 300 North Washington Street | Gettysburg, PA 17325<https://maps.google.com/?q=2453+%7C+300+North+Washington+Street+%7C+Gettysburg,+PA+17325&entry=gmail&source=g>
Phone: 717.337.6993<tel:(717)%20337-6993>
https://www.gettysburg.edu<https://www.gettysburg.edu/>

From: [hidden email]<mailto:[hidden email]> [mailto:[hidden email]] On Behalf Of Will Ezell
Sent: Thursday, January 04, 2018 12:26 PM
To: [hidden email]<mailto:[hidden email]>
Subject: Re: [dotcms] [4.1.1] 4.x Cache headaches.

I see the user code you submitted to support - I don't think it has anything to do with cache delay.  What is null is not the User but the instance of the UserCache.  Are you trying to use the UserFactoryLiferayImpl directly?    I think it has to do with using the impl factory UserFactoryLiferayImpl directly (statically) rather than calling the method through the UserAPI.  I think you should be calling:

APILocator.getUserAPI().loadUserById(userId)

On Thu, Jan 4, 2018 at 12:16 PM, Will Ezell <[hidden email]<mailto:[hidden email]>> wrote:
Nathan:

There is nothing in 4.1x that changed or is async with regards to cache invalidations- the async changes were added in the 4.2.x series and I believe (though am not certain) they can be mostly turned off by adding REINDEX_ON_SAVE_IN_SEPARATE_THREAD=false to your dotmarketing-config.  If you are saving a user and then calling for that user and getting a null, that would be a bug (unless you are in 2 transactions).  If you share your code we would be happy to take a look and either validate it or use it to write a test and fix the case.  In fact, any code where you are seeing this case in 4.1 would be a good candidate for a test.




On Thu, Jan 4, 2018 at 12:01 PM, Nathan Keiter <[hidden email]<mailto:[hidden email]>> wrote:
Just a word of warning for plugin developers moving to 4.x.  The Java APIs in 4.x are highly cache dependent. This is great for server performance in the long term. However in the immediate context of a critical business process, it is problematic.

This cache dependence makes deterministic business logic algorithms nearly impossible.

For example, if I create a new “Apple” object and save it, I expect to be able to load that “Apple” object, modify it (maybe taking a bite out of it), and save it again. Unfortunately when you go to load your newly saved “Apple”, you get a null pointer exception or some other “not found” error thrown. This is because in 4.x all the APIs hit the cache, and the cache update is so slow that your code is trying to load it before the cache is updated.

The most recent example of this we encountered is using the userAPI to create a new user. A little later in the code we try to load the newly created user via the userAPI and get a null pointer exception, even though if we look in the database, the new user has in fact been created!

This issue can bite you for anything that is cached. We’ve seen it in content, relationships, and users to name a few. Content does have an isIndexed check, but it only applies to the contentlet itself, not anything related to it, i.e. relationships, permissions etc. So even that check is not reliable in the holistic sense of “Is this thing entirely cached?”

This asynchronous “eventually available” philosophy for cache is unreliable for deterministic business processes. Hack-ish solutions include adding a Java sleep to delay your program in the hopes that the cache will catch up by the time you try to load your data, or risking an infinite loop of “is it here yet?” checks. Obviously neither is ideal.

It would be nice if there was a “no-cache” API option available so we could reliably load newly created records during critical business logic blocks. Data accuracy and immediate accessibility is of utmost importance in these cases. I can afford to take a slight performance hit to check the DB. And honestly it would probably be less of a performance hit than waiting for the cache to update.

We did not have these issues in 3.x.

Nathan I. Keiter | Lead Network Applications Programmer | Benefits Advisory Council Member | I.D.E.A Council Member
Gettysburg College | Information Technology | DataSystems
Campus Box 2453 | 300 North Washington Street | Gettysburg, PA 17325<https://maps.google.com/?q=2453+%7C+300+North+Washington+Street+%7C+Gettysburg,+PA+17325&entry=gmail&source=g>
Phone: 717.337.6993<tel:(717)%20337-6993>
https://www.gettysburg.edu<https://www.gettysburg.edu/>

--
http://dotcms.com - Open Source Java Content Management
---
You received this message because you are subscribed to the Google Groups "dotCMS User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email]<mailto:[hidden email]>.
To post to this group, send email to [hidden email]<mailto:[hidden email]>.
To view this discussion on the web visit https://groups.google.com/d/msgid/dotcms/c637e1364965480080ea2f147ef92847%40exch13b.ms.gettysburg.edu<https://groups.google.com/d/msgid/dotcms/c637e1364965480080ea2f147ef92847%40exch13b.ms.gettysburg.edu?utm_medium=email&utm_source=footer>.
For more options, visit https://groups.google.com/d/optout.



--
[https://drive.google.com/a/dotcms.com/uc?id=0B4ssYEJmR0gHdEVrSXRxMmN0b0E&export=download]


3059 Grand Avenue<https://maps.google.com/?q=3059+Grand+Avenue%0D+%0D+%0D+Suite+410&entry=gmail&source=g>
Suite 410<https://maps.google.com/?q=3059+Grand+Avenue%0D+%0D+%0D+Suite+410&entry=gmail&source=g>-B
Miami FL 33133
Main: 305-900-2001<tel:(305)%20900-2001> | Direct: 978.294.9429<tel:(978)%20294-9429>

[hidden email]<mailto:[hidden email]> | dotcms.com<http://www.dotcms.com/>

[https://origin.ih.constantcontact.com/fs172/1109987089110/img/11.png]<https://twitter.com/dotcms> [https://origin.ih.constantcontact.com/fs172/1109987089110/img/14.png] <http://www.linkedin.com/company/dotcms>  [https://origin.ih.constantcontact.com/fs172/1109987089110/img/15.png] <https://plus.google.com/+Dotcms>  [https://origin.ih.constantcontact.com/fs172/1109987089110/img/13.png] <https://www.facebook.com/dotCMS>



--
[https://drive.google.com/a/dotcms.com/uc?id=0B4ssYEJmR0gHdEVrSXRxMmN0b0E&export=download]


3059 Grand Avenue<https://maps.google.com/?q=3059+Grand+Avenue%0D+%0D+%0D+Suite+410&entry=gmail&source=g>
Suite 410<https://maps.google.com/?q=3059+Grand+Avenue%0D+%0D+%0D+Suite+410&entry=gmail&source=g>-B
Miami FL 33133
Main: 305-900-2001<tel:(305)%20900-2001> | Direct: 978.294.9429<tel:(978)%20294-9429>

[hidden email]<mailto:[hidden email]> | dotcms.com<http://www.dotcms.com/>

[https://origin.ih.constantcontact.com/fs172/1109987089110/img/11.png]<https://twitter.com/dotcms> [https://origin.ih.constantcontact.com/fs172/1109987089110/img/14.png] <http://www.linkedin.com/company/dotcms>  [https://origin.ih.constantcontact.com/fs172/1109987089110/img/15.png] <https://plus.google.com/+Dotcms>  [https://origin.ih.constantcontact.com/fs172/1109987089110/img/13.png] <https://www.facebook.com/dotCMS>
--
http://dotcms.com - Open Source Java Content Management
---
You received this message because you are subscribed to the Google Groups "dotCMS User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email]<mailto:[hidden email]>.
To post to this group, send email to [hidden email]<mailto:[hidden email]>.
To view this discussion on the web visit https://groups.google.com/d/msgid/dotcms/CAPL0VyGGDVbXoBp8OOyYV__f9O-Qc_dRBBATdUaUmD%2BXH5e2og%40mail.gmail.com<https://groups.google.com/d/msgid/dotcms/CAPL0VyGGDVbXoBp8OOyYV__f9O-Qc_dRBBATdUaUmD%2BXH5e2og%40mail.gmail.com?utm_medium=email&utm_source=footer>.
For more options, visit https://groups.google.com/d/optout.
--
http://dotcms.com - Open Source Java Content Management
---
You received this message because you are subscribed to the Google Groups "dotCMS User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email]<mailto:[hidden email]>.
To post to this group, send email to [hidden email]<mailto:[hidden email]>.
To view this discussion on the web visit https://groups.google.com/d/msgid/dotcms/e889adbfb07047de80634b693ef26a93%40exch13b.ms.gettysburg.edu<https://groups.google.com/d/msgid/dotcms/e889adbfb07047de80634b693ef26a93%40exch13b.ms.gettysburg.edu?utm_medium=email&utm_source=footer>.
For more options, visit https://groups.google.com/d/optout.
--
http://dotcms.com - Open Source Java Content Management
---
You received this message because you are subscribed to the Google Groups "dotCMS User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email]<mailto:[hidden email]>.
To post to this group, send email to [hidden email]<mailto:[hidden email]>.
To view this discussion on the web visit https://groups.google.com/d/msgid/dotcms/5f4dcd1ae1db45c9ab378ec7189d7b7a%40exch13b.ms.gettysburg.edu<https://groups.google.com/d/msgid/dotcms/5f4dcd1ae1db45c9ab378ec7189d7b7a%40exch13b.ms.gettysburg.edu?utm_medium=email&utm_source=footer>.

For more options, visit https://groups.google.com/d/optout.



--
[https://drive.google.com/a/dotcms.com/uc?id=0B4ssYEJmR0gHdEVrSXRxMmN0b0E&export=download]


3059 Grand Avenue
Suite 410-B
Miami FL 33133
Main: 305-900-2001 | Direct: 978.294.9429

[hidden email]<mailto:[hidden email]> | dotcms.com<http://www.dotcms.com/>

[https://origin.ih.constantcontact.com/fs172/1109987089110/img/11.png]<https://twitter.com/dotcms> [https://origin.ih.constantcontact.com/fs172/1109987089110/img/14.png] <http://www.linkedin.com/company/dotcms>  [https://origin.ih.constantcontact.com/fs172/1109987089110/img/15.png] <https://plus.google.com/+Dotcms>  [https://origin.ih.constantcontact.com/fs172/1109987089110/img/13.png] <https://www.facebook.com/dotCMS>
--
http://dotcms.com - Open Source Java Content Management
---
You received this message because you are subscribed to the Google Groups "dotCMS User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email]<mailto:[hidden email]>.
To post to this group, send email to [hidden email]<mailto:[hidden email]>.
To view this discussion on the web visit https://groups.google.com/d/msgid/dotcms/CAPL0VyFVpP%2BFfHvKKOGCv3C%2Br45xaBrUPu8bhKscUovvr_xZ%2BA%40mail.gmail.com<https://groups.google.com/d/msgid/dotcms/CAPL0VyFVpP%2BFfHvKKOGCv3C%2Br45xaBrUPu8bhKscUovvr_xZ%2BA%40mail.gmail.com?utm_medium=email&utm_source=footer>.
For more options, visit https://groups.google.com/d/optout.

--
http://dotcms.com - Open Source Java Content Management
---
You received this message because you are subscribed to the Google Groups "dotCMS User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email]<mailto:[hidden email]>.
To post to this group, send email to [hidden email]<mailto:[hidden email]>.
To view this discussion on the web visit https://groups.google.com/d/msgid/dotcms/ca3357b3fe34460399c3807d69fd0fa5%40exch13b.ms.gettysburg.edu<https://groups.google.com/d/msgid/dotcms/ca3357b3fe34460399c3807d69fd0fa5%40exch13b.ms.gettysburg.edu?utm_medium=email&utm_source=footer>.
For more options, visit https://groups.google.com/d/optout.

--
http://dotcms.com - Open Source Java Content Management
---
You received this message because you are subscribed to the Google Groups "dotCMS User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/dotcms/1515094138513.66485%40gettysburg.edu.
For more options, visit https://groups.google.com/d/optout.

Screen Shot 2018-01-04 at 2.23.58 PM.png (227K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

RE: [4.1.1] 4.x Cache headaches.

Nathan Keiter
I only had to start using that hack after upgrading to 4.1.1.

I've timed the delay and it's often only less than 1 second.

However my code is running faster than that.

So the thread config option might help there.

Nathan I. Keiter | Lead Network Applications Programmer | Benefits Advisory Council Member | I.D.E.A Council Member
Gettysburg College | Information Technology | DataSystems
Campus Box 2453 | 300 North Washington Street | Gettysburg, PA 17325
Phone: 717.337.6993
https://www.gettysburg.edu

-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On Behalf Of Nathan Keiter
Sent: Thursday, January 04, 2018 2:29 PM
To: [hidden email]
Subject: Re: [dotcms] [4.1.1] 4.x Cache headaches.

Here's the hack I have to use to get related content after a fresh content save due to the cache delay:


[cid:d0199436-0727-4550-bc26-207a107be322]


Nathan I. Keiter | Lead Network Applications Programmer | Benefits Advisory Council Member | I.D.E.A Council Member Gettysburg College | Information Technology | DataSystems Campus Box 2453 | 300 North Washington Street | Gettysburg, PA 17325
Phone: 717.337.6993
https://www.gettysburg.edu
________________________________
From: [hidden email] <[hidden email]> on behalf of Nathan Keiter <[hidden email]>
Sent: Thursday, January 4, 2018 2:22 PM
To: [hidden email]
Subject: RE: [dotcms] [4.1.1] 4.x Cache headaches.

Ok, I added a snippet to the ticket. The fact remains that this code worked 100% in 3.5.1, so it seems to point to an issue with 4.1.1. The delta is there.

Nathan I. Keiter | Lead Network Applications Programmer | Benefits Advisory Council Member | I.D.E.A Council Member Gettysburg College | Information Technology | DataSystems Campus Box 2453 | 300 North Washington Street | Gettysburg, PA 17325
Phone: 717.337.6993
https://www.gettysburg.edu<https://www.gettysburg.edu/>

From: [hidden email] [mailto:[hidden email]] On Behalf Of Will Ezell
Sent: Thursday, January 04, 2018 1:28 PM
To: [hidden email]
Subject: Re: [dotcms] [4.1.1] 4.x Cache headaches.

Please give us a code snippet so we can try to reproduce and write a unit test against.  If it is a code issue, we will get it fixed.

On Thu, Jan 4, 2018 at 12:51 PM, Nathan Keiter <[hidden email]<mailto:[hidden email]>> wrote:
I’ve been dealing with and researching this issue for several weeks now on two different 4.1.1 instances we have. I am very sure it’s a cache delay issue.

This does look promising: REINDEX_ON_SAVE_IN_SEPARATE_THREAD=false

Nathan I. Keiter | Lead Network Applications Programmer | Benefits Advisory Council Member | I.D.E.A Council Member Gettysburg College | Information Technology | DataSystems Campus Box 2453 | 300 North Washington Street | Gettysburg, PA 17325<https://maps.google.com/?q=2453+%7C+300+North+Washington+Street+%7C+Gettysburg,+PA+17325&entry=gmail&source=g>
Phone: 717.337.6993<tel:(717)%20337-6993>
https://www.gettysburg.edu<https://www.gettysburg.edu/>

From: [hidden email]<mailto:[hidden email]> [mailto:[hidden email]<mailto:[hidden email]>] On Behalf Of Nathan Keiter
Sent: Thursday, January 04, 2018 12:30 PM
To: [hidden email]<mailto:[hidden email]>
Subject: RE: [dotcms] [4.1.1] 4.x Cache headaches.

No, I am getting it via the APILocator higher up in the code.

I’ll continue with this on the support ticket.

However, like I said, typically this goes away by adding a java sleep for enough milliseconds…

Nathan I. Keiter | Lead Network Applications Programmer | Benefits Advisory Council Member | I.D.E.A Council Member Gettysburg College | Information Technology | DataSystems Campus Box 2453 | 300 North Washington Street | Gettysburg, PA 17325<https://maps.google.com/?q=2453+%7C+300+North+Washington+Street+%7C+Gettysburg,+PA+17325&entry=gmail&source=g>
Phone: 717.337.6993<tel:(717)%20337-6993>
https://www.gettysburg.edu<https://www.gettysburg.edu/>

From: [hidden email]<mailto:[hidden email]> [mailto:[hidden email]] On Behalf Of Will Ezell
Sent: Thursday, January 04, 2018 12:26 PM
To: [hidden email]<mailto:[hidden email]>
Subject: Re: [dotcms] [4.1.1] 4.x Cache headaches.

I see the user code you submitted to support - I don't think it has anything to do with cache delay.  What is null is not the User but the instance of the UserCache.  Are you trying to use the UserFactoryLiferayImpl directly?    I think it has to do with using the impl factory UserFactoryLiferayImpl directly (statically) rather than calling the method through the UserAPI.  I think you should be calling:

APILocator.getUserAPI().loadUserById(userId)

On Thu, Jan 4, 2018 at 12:16 PM, Will Ezell <[hidden email]<mailto:[hidden email]>> wrote:
Nathan:

There is nothing in 4.1x that changed or is async with regards to cache invalidations- the async changes were added in the 4.2.x series and I believe (though am not certain) they can be mostly turned off by adding REINDEX_ON_SAVE_IN_SEPARATE_THREAD=false to your dotmarketing-config.  If you are saving a user and then calling for that user and getting a null, that would be a bug (unless you are in 2 transactions).  If you share your code we would be happy to take a look and either validate it or use it to write a test and fix the case.  In fact, any code where you are seeing this case in 4.1 would be a good candidate for a test.




On Thu, Jan 4, 2018 at 12:01 PM, Nathan Keiter <[hidden email]<mailto:[hidden email]>> wrote:
Just a word of warning for plugin developers moving to 4.x.  The Java APIs in 4.x are highly cache dependent. This is great for server performance in the long term. However in the immediate context of a critical business process, it is problematic.

This cache dependence makes deterministic business logic algorithms nearly impossible.

For example, if I create a new “Apple” object and save it, I expect to be able to load that “Apple” object, modify it (maybe taking a bite out of it), and save it again. Unfortunately when you go to load your newly saved “Apple”, you get a null pointer exception or some other “not found” error thrown. This is because in 4.x all the APIs hit the cache, and the cache update is so slow that your code is trying to load it before the cache is updated.

The most recent example of this we encountered is using the userAPI to create a new user. A little later in the code we try to load the newly created user via the userAPI and get a null pointer exception, even though if we look in the database, the new user has in fact been created!

This issue can bite you for anything that is cached. We’ve seen it in content, relationships, and users to name a few. Content does have an isIndexed check, but it only applies to the contentlet itself, not anything related to it, i.e. relationships, permissions etc. So even that check is not reliable in the holistic sense of “Is this thing entirely cached?”

This asynchronous “eventually available” philosophy for cache is unreliable for deterministic business processes. Hack-ish solutions include adding a Java sleep to delay your program in the hopes that the cache will catch up by the time you try to load your data, or risking an infinite loop of “is it here yet?” checks. Obviously neither is ideal.

It would be nice if there was a “no-cache” API option available so we could reliably load newly created records during critical business logic blocks. Data accuracy and immediate accessibility is of utmost importance in these cases. I can afford to take a slight performance hit to check the DB. And honestly it would probably be less of a performance hit than waiting for the cache to update.

We did not have these issues in 3.x.

Nathan I. Keiter | Lead Network Applications Programmer | Benefits Advisory Council Member | I.D.E.A Council Member Gettysburg College | Information Technology | DataSystems Campus Box 2453 | 300 North Washington Street | Gettysburg, PA 17325<https://maps.google.com/?q=2453+%7C+300+North+Washington+Street+%7C+Gettysburg,+PA+17325&entry=gmail&source=g>
Phone: 717.337.6993<tel:(717)%20337-6993>
https://www.gettysburg.edu<https://www.gettysburg.edu/>

--
http://dotcms.com - Open Source Java Content Management
---
You received this message because you are subscribed to the Google Groups "dotCMS User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email]<mailto:[hidden email]>.
To post to this group, send email to [hidden email]<mailto:[hidden email]>.
To view this discussion on the web visit https://groups.google.com/d/msgid/dotcms/c637e1364965480080ea2f147ef92847%40exch13b.ms.gettysburg.edu<https://groups.google.com/d/msgid/dotcms/c637e1364965480080ea2f147ef92847%40exch13b.ms.gettysburg.edu?utm_medium=email&utm_source=footer>.
For more options, visit https://groups.google.com/d/optout.



--
[https://drive.google.com/a/dotcms.com/uc?id=0B4ssYEJmR0gHdEVrSXRxMmN0b0E&export=download]


3059 Grand Avenue<https://maps.google.com/?q=3059+Grand+Avenue%0D+%0D+%0D+Suite+410&entry=gmail&source=g>
Suite 410<https://maps.google.com/?q=3059+Grand+Avenue%0D+%0D+%0D+Suite+410&entry=gmail&source=g>-B
Miami FL 33133
Main: 305-900-2001<tel:(305)%20900-2001> | Direct: 978.294.9429<tel:(978)%20294-9429>

[hidden email]<mailto:[hidden email]> | dotcms.com<http://www.dotcms.com/>

[https://origin.ih.constantcontact.com/fs172/1109987089110/img/11.png]<https://twitter.com/dotcms> [https://origin.ih.constantcontact.com/fs172/1109987089110/img/14.png] <http://www.linkedin.com/company/dotcms>  [https://origin.ih.constantcontact.com/fs172/1109987089110/img/15.png] <https://plus.google.com/+Dotcms>  [https://origin.ih.constantcontact.com/fs172/1109987089110/img/13.png] <https://www.facebook.com/dotCMS>



--
[https://drive.google.com/a/dotcms.com/uc?id=0B4ssYEJmR0gHdEVrSXRxMmN0b0E&export=download]


3059 Grand Avenue<https://maps.google.com/?q=3059+Grand+Avenue%0D+%0D+%0D+Suite+410&entry=gmail&source=g>
Suite 410<https://maps.google.com/?q=3059+Grand+Avenue%0D+%0D+%0D+Suite+410&entry=gmail&source=g>-B
Miami FL 33133
Main: 305-900-2001<tel:(305)%20900-2001> | Direct: 978.294.9429<tel:(978)%20294-9429>

[hidden email]<mailto:[hidden email]> | dotcms.com<http://www.dotcms.com/>

[https://origin.ih.constantcontact.com/fs172/1109987089110/img/11.png]<https://twitter.com/dotcms> [https://origin.ih.constantcontact.com/fs172/1109987089110/img/14.png] <http://www.linkedin.com/company/dotcms>  [https://origin.ih.constantcontact.com/fs172/1109987089110/img/15.png] <https://plus.google.com/+Dotcms>  [https://origin.ih.constantcontact.com/fs172/1109987089110/img/13.png] <https://www.facebook.com/dotCMS>
--
http://dotcms.com - Open Source Java Content Management
---
You received this message because you are subscribed to the Google Groups "dotCMS User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email]<mailto:[hidden email]>.
To post to this group, send email to [hidden email]<mailto:[hidden email]>.
To view this discussion on the web visit https://groups.google.com/d/msgid/dotcms/CAPL0VyGGDVbXoBp8OOyYV__f9O-Qc_dRBBATdUaUmD%2BXH5e2og%40mail.gmail.com<https://groups.google.com/d/msgid/dotcms/CAPL0VyGGDVbXoBp8OOyYV__f9O-Qc_dRBBATdUaUmD%2BXH5e2og%40mail.gmail.com?utm_medium=email&utm_source=footer>.
For more options, visit https://groups.google.com/d/optout.
--
http://dotcms.com - Open Source Java Content Management
---
You received this message because you are subscribed to the Google Groups "dotCMS User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email]<mailto:[hidden email]>.
To post to this group, send email to [hidden email]<mailto:[hidden email]>.
To view this discussion on the web visit https://groups.google.com/d/msgid/dotcms/e889adbfb07047de80634b693ef26a93%40exch13b.ms.gettysburg.edu<https://groups.google.com/d/msgid/dotcms/e889adbfb07047de80634b693ef26a93%40exch13b.ms.gettysburg.edu?utm_medium=email&utm_source=footer>.
For more options, visit https://groups.google.com/d/optout.
--
http://dotcms.com - Open Source Java Content Management
---
You received this message because you are subscribed to the Google Groups "dotCMS User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email]<mailto:[hidden email]>.
To post to this group, send email to [hidden email]<mailto:[hidden email]>.
To view this discussion on the web visit https://groups.google.com/d/msgid/dotcms/5f4dcd1ae1db45c9ab378ec7189d7b7a%40exch13b.ms.gettysburg.edu<https://groups.google.com/d/msgid/dotcms/5f4dcd1ae1db45c9ab378ec7189d7b7a%40exch13b.ms.gettysburg.edu?utm_medium=email&utm_source=footer>.

For more options, visit https://groups.google.com/d/optout.



--
[https://drive.google.com/a/dotcms.com/uc?id=0B4ssYEJmR0gHdEVrSXRxMmN0b0E&export=download]


3059 Grand Avenue
Suite 410-B
Miami FL 33133
Main: 305-900-2001 | Direct: 978.294.9429

[hidden email]<mailto:[hidden email]> | dotcms.com<http://www.dotcms.com/>

[https://origin.ih.constantcontact.com/fs172/1109987089110/img/11.png]<https://twitter.com/dotcms> [https://origin.ih.constantcontact.com/fs172/1109987089110/img/14.png] <http://www.linkedin.com/company/dotcms>  [https://origin.ih.constantcontact.com/fs172/1109987089110/img/15.png] <https://plus.google.com/+Dotcms>  [https://origin.ih.constantcontact.com/fs172/1109987089110/img/13.png] <https://www.facebook.com/dotCMS>
--
http://dotcms.com - Open Source Java Content Management
---
You received this message because you are subscribed to the Google Groups "dotCMS User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email]<mailto:[hidden email]>.
To post to this group, send email to [hidden email]<mailto:[hidden email]>.
To view this discussion on the web visit https://groups.google.com/d/msgid/dotcms/CAPL0VyFVpP%2BFfHvKKOGCv3C%2Br45xaBrUPu8bhKscUovvr_xZ%2BA%40mail.gmail.com<https://groups.google.com/d/msgid/dotcms/CAPL0VyFVpP%2BFfHvKKOGCv3C%2Br45xaBrUPu8bhKscUovvr_xZ%2BA%40mail.gmail.com?utm_medium=email&utm_source=footer>.
For more options, visit https://groups.google.com/d/optout.

--
http://dotcms.com - Open Source Java Content Management
---
You received this message because you are subscribed to the Google Groups "dotCMS User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email]<mailto:[hidden email]>.
To post to this group, send email to [hidden email]<mailto:[hidden email]>.
To view this discussion on the web visit https://groups.google.com/d/msgid/dotcms/ca3357b3fe34460399c3807d69fd0fa5%40exch13b.ms.gettysburg.edu<https://groups.google.com/d/msgid/dotcms/ca3357b3fe34460399c3807d69fd0fa5%40exch13b.ms.gettysburg.edu?utm_medium=email&utm_source=footer>.
For more options, visit https://groups.google.com/d/optout.

--
http://dotcms.com - Open Source Java Content Management
---
You received this message because you are subscribed to the Google Groups "dotCMS User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/dotcms/1515094138513.66485%40gettysburg.edu.
For more options, visit https://groups.google.com/d/optout.

--
http://dotcms.com - Open Source Java Content Management
---
You received this message because you are subscribed to the Google Groups "dotCMS User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/dotcms/311307b6084546b0949e0409ff300dac%40exch13b.ms.gettysburg.edu.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

RE: [4.1.1] 4.x Cache headaches.

Nathan Keiter
Of course there's other times where the delay is several seconds. It's like Schrödinger's cat, you never know until you try to look. The 10,000 try loop is usually long enough to find it.

Nathan I. Keiter | Lead Network Applications Programmer | Benefits Advisory Council Member | I.D.E.A Council Member
Gettysburg College | Information Technology | DataSystems
Campus Box 2453 | 300 North Washington Street | Gettysburg, PA 17325
Phone: 717.337.6993
https://www.gettysburg.edu


-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On Behalf Of Nathan Keiter
Sent: Thursday, January 04, 2018 2:43 PM
To: [hidden email]
Subject: RE: [dotcms] [4.1.1] 4.x Cache headaches.

I only had to start using that hack after upgrading to 4.1.1.

I've timed the delay and it's often only less than 1 second.

However my code is running faster than that.

So the thread config option might help there.

Nathan I. Keiter | Lead Network Applications Programmer | Benefits Advisory Council Member | I.D.E.A Council Member Gettysburg College | Information Technology | DataSystems Campus Box 2453 | 300 North Washington Street | Gettysburg, PA 17325
Phone: 717.337.6993
https://www.gettysburg.edu

-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On Behalf Of Nathan Keiter
Sent: Thursday, January 04, 2018 2:29 PM
To: [hidden email]
Subject: Re: [dotcms] [4.1.1] 4.x Cache headaches.

Here's the hack I have to use to get related content after a fresh content save due to the cache delay:


[cid:d0199436-0727-4550-bc26-207a107be322]


Nathan I. Keiter | Lead Network Applications Programmer | Benefits Advisory Council Member | I.D.E.A Council Member Gettysburg College | Information Technology | DataSystems Campus Box 2453 | 300 North Washington Street | Gettysburg, PA 17325
Phone: 717.337.6993
https://www.gettysburg.edu
________________________________
From: [hidden email] <[hidden email]> on behalf of Nathan Keiter <[hidden email]>
Sent: Thursday, January 4, 2018 2:22 PM
To: [hidden email]
Subject: RE: [dotcms] [4.1.1] 4.x Cache headaches.

Ok, I added a snippet to the ticket. The fact remains that this code worked 100% in 3.5.1, so it seems to point to an issue with 4.1.1. The delta is there.

Nathan I. Keiter | Lead Network Applications Programmer | Benefits Advisory Council Member | I.D.E.A Council Member Gettysburg College | Information Technology | DataSystems Campus Box 2453 | 300 North Washington Street | Gettysburg, PA 17325
Phone: 717.337.6993
https://www.gettysburg.edu<https://www.gettysburg.edu/>

From: [hidden email] [mailto:[hidden email]] On Behalf Of Will Ezell
Sent: Thursday, January 04, 2018 1:28 PM
To: [hidden email]
Subject: Re: [dotcms] [4.1.1] 4.x Cache headaches.

Please give us a code snippet so we can try to reproduce and write a unit test against.  If it is a code issue, we will get it fixed.

On Thu, Jan 4, 2018 at 12:51 PM, Nathan Keiter <[hidden email]<mailto:[hidden email]>> wrote:
I’ve been dealing with and researching this issue for several weeks now on two different 4.1.1 instances we have. I am very sure it’s a cache delay issue.

This does look promising: REINDEX_ON_SAVE_IN_SEPARATE_THREAD=false

Nathan I. Keiter | Lead Network Applications Programmer | Benefits Advisory Council Member | I.D.E.A Council Member Gettysburg College | Information Technology | DataSystems Campus Box 2453 | 300 North Washington Street | Gettysburg, PA 17325<https://maps.google.com/?q=2453+%7C+300+North+Washington+Street+%7C+Gettysburg,+PA+17325&entry=gmail&source=g>
Phone: 717.337.6993<tel:(717)%20337-6993>
https://www.gettysburg.edu<https://www.gettysburg.edu/>

From: [hidden email]<mailto:[hidden email]> [mailto:[hidden email]<mailto:[hidden email]>] On Behalf Of Nathan Keiter
Sent: Thursday, January 04, 2018 12:30 PM
To: [hidden email]<mailto:[hidden email]>
Subject: RE: [dotcms] [4.1.1] 4.x Cache headaches.

No, I am getting it via the APILocator higher up in the code.

I’ll continue with this on the support ticket.

However, like I said, typically this goes away by adding a java sleep for enough milliseconds…

Nathan I. Keiter | Lead Network Applications Programmer | Benefits Advisory Council Member | I.D.E.A Council Member Gettysburg College | Information Technology | DataSystems Campus Box 2453 | 300 North Washington Street | Gettysburg, PA 17325<https://maps.google.com/?q=2453+%7C+300+North+Washington+Street+%7C+Gettysburg,+PA+17325&entry=gmail&source=g>
Phone: 717.337.6993<tel:(717)%20337-6993>
https://www.gettysburg.edu<https://www.gettysburg.edu/>

From: [hidden email]<mailto:[hidden email]> [mailto:[hidden email]] On Behalf Of Will Ezell
Sent: Thursday, January 04, 2018 12:26 PM
To: [hidden email]<mailto:[hidden email]>
Subject: Re: [dotcms] [4.1.1] 4.x Cache headaches.

I see the user code you submitted to support - I don't think it has anything to do with cache delay.  What is null is not the User but the instance of the UserCache.  Are you trying to use the UserFactoryLiferayImpl directly?    I think it has to do with using the impl factory UserFactoryLiferayImpl directly (statically) rather than calling the method through the UserAPI.  I think you should be calling:

APILocator.getUserAPI().loadUserById(userId)

On Thu, Jan 4, 2018 at 12:16 PM, Will Ezell <[hidden email]<mailto:[hidden email]>> wrote:
Nathan:

There is nothing in 4.1x that changed or is async with regards to cache invalidations- the async changes were added in the 4.2.x series and I believe (though am not certain) they can be mostly turned off by adding REINDEX_ON_SAVE_IN_SEPARATE_THREAD=false to your dotmarketing-config.  If you are saving a user and then calling for that user and getting a null, that would be a bug (unless you are in 2 transactions).  If you share your code we would be happy to take a look and either validate it or use it to write a test and fix the case.  In fact, any code where you are seeing this case in 4.1 would be a good candidate for a test.




On Thu, Jan 4, 2018 at 12:01 PM, Nathan Keiter <[hidden email]<mailto:[hidden email]>> wrote:
Just a word of warning for plugin developers moving to 4.x.  The Java APIs in 4.x are highly cache dependent. This is great for server performance in the long term. However in the immediate context of a critical business process, it is problematic.

This cache dependence makes deterministic business logic algorithms nearly impossible.

For example, if I create a new “Apple” object and save it, I expect to be able to load that “Apple” object, modify it (maybe taking a bite out of it), and save it again. Unfortunately when you go to load your newly saved “Apple”, you get a null pointer exception or some other “not found” error thrown. This is because in 4.x all the APIs hit the cache, and the cache update is so slow that your code is trying to load it before the cache is updated.

The most recent example of this we encountered is using the userAPI to create a new user. A little later in the code we try to load the newly created user via the userAPI and get a null pointer exception, even though if we look in the database, the new user has in fact been created!

This issue can bite you for anything that is cached. We’ve seen it in content, relationships, and users to name a few. Content does have an isIndexed check, but it only applies to the contentlet itself, not anything related to it, i.e. relationships, permissions etc. So even that check is not reliable in the holistic sense of “Is this thing entirely cached?”

This asynchronous “eventually available” philosophy for cache is unreliable for deterministic business processes. Hack-ish solutions include adding a Java sleep to delay your program in the hopes that the cache will catch up by the time you try to load your data, or risking an infinite loop of “is it here yet?” checks. Obviously neither is ideal.

It would be nice if there was a “no-cache” API option available so we could reliably load newly created records during critical business logic blocks. Data accuracy and immediate accessibility is of utmost importance in these cases. I can afford to take a slight performance hit to check the DB. And honestly it would probably be less of a performance hit than waiting for the cache to update.

We did not have these issues in 3.x.

Nathan I. Keiter | Lead Network Applications Programmer | Benefits Advisory Council Member | I.D.E.A Council Member Gettysburg College | Information Technology | DataSystems Campus Box 2453 | 300 North Washington Street | Gettysburg, PA 17325<https://maps.google.com/?q=2453+%7C+300+North+Washington+Street+%7C+Gettysburg,+PA+17325&entry=gmail&source=g>
Phone: 717.337.6993<tel:(717)%20337-6993>
https://www.gettysburg.edu<https://www.gettysburg.edu/>

--
http://dotcms.com - Open Source Java Content Management
---
You received this message because you are subscribed to the Google Groups "dotCMS User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email]<mailto:[hidden email]>.
To post to this group, send email to [hidden email]<mailto:[hidden email]>.
To view this discussion on the web visit https://groups.google.com/d/msgid/dotcms/c637e1364965480080ea2f147ef92847%40exch13b.ms.gettysburg.edu<https://groups.google.com/d/msgid/dotcms/c637e1364965480080ea2f147ef92847%40exch13b.ms.gettysburg.edu?utm_medium=email&utm_source=footer>.
For more options, visit https://groups.google.com/d/optout.



--
[https://drive.google.com/a/dotcms.com/uc?id=0B4ssYEJmR0gHdEVrSXRxMmN0b0E&export=download]


3059 Grand Avenue<https://maps.google.com/?q=3059+Grand+Avenue%0D+%0D+%0D+Suite+410&entry=gmail&source=g>
Suite 410<https://maps.google.com/?q=3059+Grand+Avenue%0D+%0D+%0D+Suite+410&entry=gmail&source=g>-B
Miami FL 33133
Main: 305-900-2001<tel:(305)%20900-2001> | Direct: 978.294.9429<tel:(978)%20294-9429>

[hidden email]<mailto:[hidden email]> | dotcms.com<http://www.dotcms.com/>

[https://origin.ih.constantcontact.com/fs172/1109987089110/img/11.png]<https://twitter.com/dotcms> [https://origin.ih.constantcontact.com/fs172/1109987089110/img/14.png] <http://www.linkedin.com/company/dotcms>  [https://origin.ih.constantcontact.com/fs172/1109987089110/img/15.png] <https://plus.google.com/+Dotcms>  [https://origin.ih.constantcontact.com/fs172/1109987089110/img/13.png] <https://www.facebook.com/dotCMS>



--
[https://drive.google.com/a/dotcms.com/uc?id=0B4ssYEJmR0gHdEVrSXRxMmN0b0E&export=download]


3059 Grand Avenue<https://maps.google.com/?q=3059+Grand+Avenue%0D+%0D+%0D+Suite+410&entry=gmail&source=g>
Suite 410<https://maps.google.com/?q=3059+Grand+Avenue%0D+%0D+%0D+Suite+410&entry=gmail&source=g>-B
Miami FL 33133
Main: 305-900-2001<tel:(305)%20900-2001> | Direct: 978.294.9429<tel:(978)%20294-9429>

[hidden email]<mailto:[hidden email]> | dotcms.com<http://www.dotcms.com/>

[https://origin.ih.constantcontact.com/fs172/1109987089110/img/11.png]<https://twitter.com/dotcms> [https://origin.ih.constantcontact.com/fs172/1109987089110/img/14.png] <http://www.linkedin.com/company/dotcms>  [https://origin.ih.constantcontact.com/fs172/1109987089110/img/15.png] <https://plus.google.com/+Dotcms>  [https://origin.ih.constantcontact.com/fs172/1109987089110/img/13.png] <https://www.facebook.com/dotCMS>
--
http://dotcms.com - Open Source Java Content Management
---
You received this message because you are subscribed to the Google Groups "dotCMS User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email]<mailto:[hidden email]>.
To post to this group, send email to [hidden email]<mailto:[hidden email]>.
To view this discussion on the web visit https://groups.google.com/d/msgid/dotcms/CAPL0VyGGDVbXoBp8OOyYV__f9O-Qc_dRBBATdUaUmD%2BXH5e2og%40mail.gmail.com<https://groups.google.com/d/msgid/dotcms/CAPL0VyGGDVbXoBp8OOyYV__f9O-Qc_dRBBATdUaUmD%2BXH5e2og%40mail.gmail.com?utm_medium=email&utm_source=footer>.
For more options, visit https://groups.google.com/d/optout.
--
http://dotcms.com - Open Source Java Content Management
---
You received this message because you are subscribed to the Google Groups "dotCMS User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email]<mailto:[hidden email]>.
To post to this group, send email to [hidden email]<mailto:[hidden email]>.
To view this discussion on the web visit https://groups.google.com/d/msgid/dotcms/e889adbfb07047de80634b693ef26a93%40exch13b.ms.gettysburg.edu<https://groups.google.com/d/msgid/dotcms/e889adbfb07047de80634b693ef26a93%40exch13b.ms.gettysburg.edu?utm_medium=email&utm_source=footer>.
For more options, visit https://groups.google.com/d/optout.
--
http://dotcms.com - Open Source Java Content Management
---
You received this message because you are subscribed to the Google Groups "dotCMS User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email]<mailto:[hidden email]>.
To post to this group, send email to [hidden email]<mailto:[hidden email]>.
To view this discussion on the web visit https://groups.google.com/d/msgid/dotcms/5f4dcd1ae1db45c9ab378ec7189d7b7a%40exch13b.ms.gettysburg.edu<https://groups.google.com/d/msgid/dotcms/5f4dcd1ae1db45c9ab378ec7189d7b7a%40exch13b.ms.gettysburg.edu?utm_medium=email&utm_source=footer>.

For more options, visit https://groups.google.com/d/optout.



--
[https://drive.google.com/a/dotcms.com/uc?id=0B4ssYEJmR0gHdEVrSXRxMmN0b0E&export=download]


3059 Grand Avenue
Suite 410-B
Miami FL 33133
Main: 305-900-2001 | Direct: 978.294.9429

[hidden email]<mailto:[hidden email]> | dotcms.com<http://www.dotcms.com/>

[https://origin.ih.constantcontact.com/fs172/1109987089110/img/11.png]<https://twitter.com/dotcms> [https://origin.ih.constantcontact.com/fs172/1109987089110/img/14.png] <http://www.linkedin.com/company/dotcms>  [https://origin.ih.constantcontact.com/fs172/1109987089110/img/15.png] <https://plus.google.com/+Dotcms>  [https://origin.ih.constantcontact.com/fs172/1109987089110/img/13.png] <https://www.facebook.com/dotCMS>
--
http://dotcms.com - Open Source Java Content Management
---
You received this message because you are subscribed to the Google Groups "dotCMS User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email]<mailto:[hidden email]>.
To post to this group, send email to [hidden email]<mailto:[hidden email]>.
To view this discussion on the web visit https://groups.google.com/d/msgid/dotcms/CAPL0VyFVpP%2BFfHvKKOGCv3C%2Br45xaBrUPu8bhKscUovvr_xZ%2BA%40mail.gmail.com<https://groups.google.com/d/msgid/dotcms/CAPL0VyFVpP%2BFfHvKKOGCv3C%2Br45xaBrUPu8bhKscUovvr_xZ%2BA%40mail.gmail.com?utm_medium=email&utm_source=footer>.
For more options, visit https://groups.google.com/d/optout.

--
http://dotcms.com - Open Source Java Content Management
---
You received this message because you are subscribed to the Google Groups "dotCMS User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email]<mailto:[hidden email]>.
To post to this group, send email to [hidden email]<mailto:[hidden email]>.
To view this discussion on the web visit https://groups.google.com/d/msgid/dotcms/ca3357b3fe34460399c3807d69fd0fa5%40exch13b.ms.gettysburg.edu<https://groups.google.com/d/msgid/dotcms/ca3357b3fe34460399c3807d69fd0fa5%40exch13b.ms.gettysburg.edu?utm_medium=email&utm_source=footer>.
For more options, visit https://groups.google.com/d/optout.

--
http://dotcms.com - Open Source Java Content Management
---
You received this message because you are subscribed to the Google Groups "dotCMS User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/dotcms/1515094138513.66485%40gettysburg.edu.
For more options, visit https://groups.google.com/d/optout.

--
http://dotcms.com - Open Source Java Content Management
---
You received this message because you are subscribed to the Google Groups "dotCMS User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/dotcms/311307b6084546b0949e0409ff300dac%40exch13b.ms.gettysburg.edu.
For more options, visit https://groups.google.com/d/optout.

--
http://dotcms.com - Open Source Java Content Management
---
You received this message because you are subscribed to the Google Groups "dotCMS User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/dotcms/3f8093460cfb4b32ad736fb6e2eb6360%40exch13b.ms.gettysburg.edu.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

RE: [4.1.1] 4.x Cache headaches.

Nathan Keiter
The "isInodeIndexed" call is actually useless here and should be removed. Because it's actually the relationship we need to be indexed, not the content. It doesn't hurt to be there though, just haven't had time to remove it yet.

Nathan I. Keiter | Lead Network Applications Programmer | Benefits Advisory Council Member | I.D.E.A Council Member
Gettysburg College | Information Technology | DataSystems
Campus Box 2453 | 300 North Washington Street | Gettysburg, PA 17325
Phone: 717.337.6993
https://www.gettysburg.edu


-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On Behalf Of Nathan Keiter
Sent: Thursday, January 04, 2018 2:47 PM
To: [hidden email]
Subject: RE: [dotcms] [4.1.1] 4.x Cache headaches.

Of course there's other times where the delay is several seconds. It's like Schrödinger's cat, you never know until you try to look. The 10,000 try loop is usually long enough to find it.

Nathan I. Keiter | Lead Network Applications Programmer | Benefits Advisory Council Member | I.D.E.A Council Member Gettysburg College | Information Technology | DataSystems Campus Box 2453 | 300 North Washington Street | Gettysburg, PA 17325
Phone: 717.337.6993
https://www.gettysburg.edu


-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On Behalf Of Nathan Keiter
Sent: Thursday, January 04, 2018 2:43 PM
To: [hidden email]
Subject: RE: [dotcms] [4.1.1] 4.x Cache headaches.

I only had to start using that hack after upgrading to 4.1.1.

I've timed the delay and it's often only less than 1 second.

However my code is running faster than that.

So the thread config option might help there.

Nathan I. Keiter | Lead Network Applications Programmer | Benefits Advisory Council Member | I.D.E.A Council Member Gettysburg College | Information Technology | DataSystems Campus Box 2453 | 300 North Washington Street | Gettysburg, PA 17325
Phone: 717.337.6993
https://www.gettysburg.edu

-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On Behalf Of Nathan Keiter
Sent: Thursday, January 04, 2018 2:29 PM
To: [hidden email]
Subject: Re: [dotcms] [4.1.1] 4.x Cache headaches.

Here's the hack I have to use to get related content after a fresh content save due to the cache delay:


[cid:d0199436-0727-4550-bc26-207a107be322]


Nathan I. Keiter | Lead Network Applications Programmer | Benefits Advisory Council Member | I.D.E.A Council Member Gettysburg College | Information Technology | DataSystems Campus Box 2453 | 300 North Washington Street | Gettysburg, PA 17325
Phone: 717.337.6993
https://www.gettysburg.edu
________________________________
From: [hidden email] <[hidden email]> on behalf of Nathan Keiter <[hidden email]>
Sent: Thursday, January 4, 2018 2:22 PM
To: [hidden email]
Subject: RE: [dotcms] [4.1.1] 4.x Cache headaches.

Ok, I added a snippet to the ticket. The fact remains that this code worked 100% in 3.5.1, so it seems to point to an issue with 4.1.1. The delta is there.

Nathan I. Keiter | Lead Network Applications Programmer | Benefits Advisory Council Member | I.D.E.A Council Member Gettysburg College | Information Technology | DataSystems Campus Box 2453 | 300 North Washington Street | Gettysburg, PA 17325
Phone: 717.337.6993
https://www.gettysburg.edu<https://www.gettysburg.edu/>

From: [hidden email] [mailto:[hidden email]] On Behalf Of Will Ezell
Sent: Thursday, January 04, 2018 1:28 PM
To: [hidden email]
Subject: Re: [dotcms] [4.1.1] 4.x Cache headaches.

Please give us a code snippet so we can try to reproduce and write a unit test against.  If it is a code issue, we will get it fixed.

On Thu, Jan 4, 2018 at 12:51 PM, Nathan Keiter <[hidden email]<mailto:[hidden email]>> wrote:
I’ve been dealing with and researching this issue for several weeks now on two different 4.1.1 instances we have. I am very sure it’s a cache delay issue.

This does look promising: REINDEX_ON_SAVE_IN_SEPARATE_THREAD=false

Nathan I. Keiter | Lead Network Applications Programmer | Benefits Advisory Council Member | I.D.E.A Council Member Gettysburg College | Information Technology | DataSystems Campus Box 2453 | 300 North Washington Street | Gettysburg, PA 17325<https://maps.google.com/?q=2453+%7C+300+North+Washington+Street+%7C+Gettysburg,+PA+17325&entry=gmail&source=g>
Phone: 717.337.6993<tel:(717)%20337-6993>
https://www.gettysburg.edu<https://www.gettysburg.edu/>

From: [hidden email]<mailto:[hidden email]> [mailto:[hidden email]<mailto:[hidden email]>] On Behalf Of Nathan Keiter
Sent: Thursday, January 04, 2018 12:30 PM
To: [hidden email]<mailto:[hidden email]>
Subject: RE: [dotcms] [4.1.1] 4.x Cache headaches.

No, I am getting it via the APILocator higher up in the code.

I’ll continue with this on the support ticket.

However, like I said, typically this goes away by adding a java sleep for enough milliseconds…

Nathan I. Keiter | Lead Network Applications Programmer | Benefits Advisory Council Member | I.D.E.A Council Member Gettysburg College | Information Technology | DataSystems Campus Box 2453 | 300 North Washington Street | Gettysburg, PA 17325<https://maps.google.com/?q=2453+%7C+300+North+Washington+Street+%7C+Gettysburg,+PA+17325&entry=gmail&source=g>
Phone: 717.337.6993<tel:(717)%20337-6993>
https://www.gettysburg.edu<https://www.gettysburg.edu/>

From: [hidden email]<mailto:[hidden email]> [mailto:[hidden email]] On Behalf Of Will Ezell
Sent: Thursday, January 04, 2018 12:26 PM
To: [hidden email]<mailto:[hidden email]>
Subject: Re: [dotcms] [4.1.1] 4.x Cache headaches.

I see the user code you submitted to support - I don't think it has anything to do with cache delay.  What is null is not the User but the instance of the UserCache.  Are you trying to use the UserFactoryLiferayImpl directly?    I think it has to do with using the impl factory UserFactoryLiferayImpl directly (statically) rather than calling the method through the UserAPI.  I think you should be calling:

APILocator.getUserAPI().loadUserById(userId)

On Thu, Jan 4, 2018 at 12:16 PM, Will Ezell <[hidden email]<mailto:[hidden email]>> wrote:
Nathan:

There is nothing in 4.1x that changed or is async with regards to cache invalidations- the async changes were added in the 4.2.x series and I believe (though am not certain) they can be mostly turned off by adding REINDEX_ON_SAVE_IN_SEPARATE_THREAD=false to your dotmarketing-config.  If you are saving a user and then calling for that user and getting a null, that would be a bug (unless you are in 2 transactions).  If you share your code we would be happy to take a look and either validate it or use it to write a test and fix the case.  In fact, any code where you are seeing this case in 4.1 would be a good candidate for a test.




On Thu, Jan 4, 2018 at 12:01 PM, Nathan Keiter <[hidden email]<mailto:[hidden email]>> wrote:
Just a word of warning for plugin developers moving to 4.x.  The Java APIs in 4.x are highly cache dependent. This is great for server performance in the long term. However in the immediate context of a critical business process, it is problematic.

This cache dependence makes deterministic business logic algorithms nearly impossible.

For example, if I create a new “Apple” object and save it, I expect to be able to load that “Apple” object, modify it (maybe taking a bite out of it), and save it again. Unfortunately when you go to load your newly saved “Apple”, you get a null pointer exception or some other “not found” error thrown. This is because in 4.x all the APIs hit the cache, and the cache update is so slow that your code is trying to load it before the cache is updated.

The most recent example of this we encountered is using the userAPI to create a new user. A little later in the code we try to load the newly created user via the userAPI and get a null pointer exception, even though if we look in the database, the new user has in fact been created!

This issue can bite you for anything that is cached. We’ve seen it in content, relationships, and users to name a few. Content does have an isIndexed check, but it only applies to the contentlet itself, not anything related to it, i.e. relationships, permissions etc. So even that check is not reliable in the holistic sense of “Is this thing entirely cached?”

This asynchronous “eventually available” philosophy for cache is unreliable for deterministic business processes. Hack-ish solutions include adding a Java sleep to delay your program in the hopes that the cache will catch up by the time you try to load your data, or risking an infinite loop of “is it here yet?” checks. Obviously neither is ideal.

It would be nice if there was a “no-cache” API option available so we could reliably load newly created records during critical business logic blocks. Data accuracy and immediate accessibility is of utmost importance in these cases. I can afford to take a slight performance hit to check the DB. And honestly it would probably be less of a performance hit than waiting for the cache to update.

We did not have these issues in 3.x.

Nathan I. Keiter | Lead Network Applications Programmer | Benefits Advisory Council Member | I.D.E.A Council Member Gettysburg College | Information Technology | DataSystems Campus Box 2453 | 300 North Washington Street | Gettysburg, PA 17325<https://maps.google.com/?q=2453+%7C+300+North+Washington+Street+%7C+Gettysburg,+PA+17325&entry=gmail&source=g>
Phone: 717.337.6993<tel:(717)%20337-6993>
https://www.gettysburg.edu<https://www.gettysburg.edu/>

--
http://dotcms.com - Open Source Java Content Management
---
You received this message because you are subscribed to the Google Groups "dotCMS User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email]<mailto:[hidden email]>.
To post to this group, send email to [hidden email]<mailto:[hidden email]>.
To view this discussion on the web visit https://groups.google.com/d/msgid/dotcms/c637e1364965480080ea2f147ef92847%40exch13b.ms.gettysburg.edu<https://groups.google.com/d/msgid/dotcms/c637e1364965480080ea2f147ef92847%40exch13b.ms.gettysburg.edu?utm_medium=email&utm_source=footer>.
For more options, visit https://groups.google.com/d/optout.



--
[https://drive.google.com/a/dotcms.com/uc?id=0B4ssYEJmR0gHdEVrSXRxMmN0b0E&export=download]


3059 Grand Avenue<https://maps.google.com/?q=3059+Grand+Avenue%0D+%0D+%0D+Suite+410&entry=gmail&source=g>
Suite 410<https://maps.google.com/?q=3059+Grand+Avenue%0D+%0D+%0D+Suite+410&entry=gmail&source=g>-B
Miami FL 33133
Main: 305-900-2001<tel:(305)%20900-2001> | Direct: 978.294.9429<tel:(978)%20294-9429>

[hidden email]<mailto:[hidden email]> | dotcms.com<http://www.dotcms.com/>

[https://origin.ih.constantcontact.com/fs172/1109987089110/img/11.png]<https://twitter.com/dotcms> [https://origin.ih.constantcontact.com/fs172/1109987089110/img/14.png] <http://www.linkedin.com/company/dotcms>  [https://origin.ih.constantcontact.com/fs172/1109987089110/img/15.png] <https://plus.google.com/+Dotcms>  [https://origin.ih.constantcontact.com/fs172/1109987089110/img/13.png] <https://www.facebook.com/dotCMS>



--
[https://drive.google.com/a/dotcms.com/uc?id=0B4ssYEJmR0gHdEVrSXRxMmN0b0E&export=download]


3059 Grand Avenue<https://maps.google.com/?q=3059+Grand+Avenue%0D+%0D+%0D+Suite+410&entry=gmail&source=g>
Suite 410<https://maps.google.com/?q=3059+Grand+Avenue%0D+%0D+%0D+Suite+410&entry=gmail&source=g>-B
Miami FL 33133
Main: 305-900-2001<tel:(305)%20900-2001> | Direct: 978.294.9429<tel:(978)%20294-9429>

[hidden email]<mailto:[hidden email]> | dotcms.com<http://www.dotcms.com/>

[https://origin.ih.constantcontact.com/fs172/1109987089110/img/11.png]<https://twitter.com/dotcms> [https://origin.ih.constantcontact.com/fs172/1109987089110/img/14.png] <http://www.linkedin.com/company/dotcms>  [https://origin.ih.constantcontact.com/fs172/1109987089110/img/15.png] <https://plus.google.com/+Dotcms>  [https://origin.ih.constantcontact.com/fs172/1109987089110/img/13.png] <https://www.facebook.com/dotCMS>
--
http://dotcms.com - Open Source Java Content Management
---
You received this message because you are subscribed to the Google Groups "dotCMS User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email]<mailto:[hidden email]>.
To post to this group, send email to [hidden email]<mailto:[hidden email]>.
To view this discussion on the web visit https://groups.google.com/d/msgid/dotcms/CAPL0VyGGDVbXoBp8OOyYV__f9O-Qc_dRBBATdUaUmD%2BXH5e2og%40mail.gmail.com<https://groups.google.com/d/msgid/dotcms/CAPL0VyGGDVbXoBp8OOyYV__f9O-Qc_dRBBATdUaUmD%2BXH5e2og%40mail.gmail.com?utm_medium=email&utm_source=footer>.
For more options, visit https://groups.google.com/d/optout.
--
http://dotcms.com - Open Source Java Content Management
---
You received this message because you are subscribed to the Google Groups "dotCMS User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email]<mailto:[hidden email]>.
To post to this group, send email to [hidden email]<mailto:[hidden email]>.
To view this discussion on the web visit https://groups.google.com/d/msgid/dotcms/e889adbfb07047de80634b693ef26a93%40exch13b.ms.gettysburg.edu<https://groups.google.com/d/msgid/dotcms/e889adbfb07047de80634b693ef26a93%40exch13b.ms.gettysburg.edu?utm_medium=email&utm_source=footer>.
For more options, visit https://groups.google.com/d/optout.
--
http://dotcms.com - Open Source Java Content Management
---
You received this message because you are subscribed to the Google Groups "dotCMS User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email]<mailto:[hidden email]>.
To post to this group, send email to [hidden email]<mailto:[hidden email]>.
To view this discussion on the web visit https://groups.google.com/d/msgid/dotcms/5f4dcd1ae1db45c9ab378ec7189d7b7a%40exch13b.ms.gettysburg.edu<https://groups.google.com/d/msgid/dotcms/5f4dcd1ae1db45c9ab378ec7189d7b7a%40exch13b.ms.gettysburg.edu?utm_medium=email&utm_source=footer>.

For more options, visit https://groups.google.com/d/optout.



--
[https://drive.google.com/a/dotcms.com/uc?id=0B4ssYEJmR0gHdEVrSXRxMmN0b0E&export=download]


3059 Grand Avenue
Suite 410-B
Miami FL 33133
Main: 305-900-2001 | Direct: 978.294.9429

[hidden email]<mailto:[hidden email]> | dotcms.com<http://www.dotcms.com/>

[https://origin.ih.constantcontact.com/fs172/1109987089110/img/11.png]<https://twitter.com/dotcms> [https://origin.ih.constantcontact.com/fs172/1109987089110/img/14.png] <http://www.linkedin.com/company/dotcms>  [https://origin.ih.constantcontact.com/fs172/1109987089110/img/15.png] <https://plus.google.com/+Dotcms>  [https://origin.ih.constantcontact.com/fs172/1109987089110/img/13.png] <https://www.facebook.com/dotCMS>
--
http://dotcms.com - Open Source Java Content Management
---
You received this message because you are subscribed to the Google Groups "dotCMS User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email]<mailto:[hidden email]>.
To post to this group, send email to [hidden email]<mailto:[hidden email]>.
To view this discussion on the web visit https://groups.google.com/d/msgid/dotcms/CAPL0VyFVpP%2BFfHvKKOGCv3C%2Br45xaBrUPu8bhKscUovvr_xZ%2BA%40mail.gmail.com<https://groups.google.com/d/msgid/dotcms/CAPL0VyFVpP%2BFfHvKKOGCv3C%2Br45xaBrUPu8bhKscUovvr_xZ%2BA%40mail.gmail.com?utm_medium=email&utm_source=footer>.
For more options, visit https://groups.google.com/d/optout.

--
http://dotcms.com - Open Source Java Content Management
---
You received this message because you are subscribed to the Google Groups "dotCMS User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email]<mailto:[hidden email]>.
To post to this group, send email to [hidden email]<mailto:[hidden email]>.
To view this discussion on the web visit https://groups.google.com/d/msgid/dotcms/ca3357b3fe34460399c3807d69fd0fa5%40exch13b.ms.gettysburg.edu<https://groups.google.com/d/msgid/dotcms/ca3357b3fe34460399c3807d69fd0fa5%40exch13b.ms.gettysburg.edu?utm_medium=email&utm_source=footer>.
For more options, visit https://groups.google.com/d/optout.

--
http://dotcms.com - Open Source Java Content Management
---
You received this message because you are subscribed to the Google Groups "dotCMS User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/dotcms/1515094138513.66485%40gettysburg.edu.
For more options, visit https://groups.google.com/d/optout.

--
http://dotcms.com - Open Source Java Content Management
---
You received this message because you are subscribed to the Google Groups "dotCMS User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/dotcms/311307b6084546b0949e0409ff300dac%40exch13b.ms.gettysburg.edu.
For more options, visit https://groups.google.com/d/optout.

--
http://dotcms.com - Open Source Java Content Management
---
You received this message because you are subscribed to the Google Groups "dotCMS User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/dotcms/3f8093460cfb4b32ad736fb6e2eb6360%40exch13b.ms.gettysburg.edu.
For more options, visit https://groups.google.com/d/optout.

--
http://dotcms.com - Open Source Java Content Management
---
You received this message because you are subscribed to the Google Groups "dotCMS User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/dotcms/2b06b5819e904d60a5d81fc05c8910f8%40exch13b.ms.gettysburg.edu.
For more options, visit https://groups.google.com/d/optout.