Alpha DevCon 2018
Results 1 to 27 of 27

Thread: Server problems

  1. #1
    Moderator
    Real Name
    Steve Workings
    Join Date
    Apr 2000
    Location
    The Dreaded Chair
    Posts
    5,516

    Default Server problems

    One of my applications has two servers, let's call them S1 and S2. A few days ago, for no particular reason I can think of, the A5 results in S2 started to show problems. For example:

    - Both servers send emails through a work queue processed by a separate runtime instance. But S2's emails are arriving with all the CSS HTML showing in the recipients' email readers. This seems limited to one or two particular email notices.

    - Both servers make the same API call to an external service. S1 is still doing fine, but S2 is not completing (I have yet to walk through it, figure out exactly what/where/why)

    - I have function List_ABC(Type as C) that's worked for years and still does on S1. But on S2 it fails and records a "Too many parameters" message in the Xbasic Error Log. There is *nothing* wrong with this function. I have several other functions failing and returning the same message.

    - A new error message in the logs I've not seen before and don't know what it means, but has my attention: "Invalid AE error code"

    These are just representative problems, not a complete list.

    I've republished the offending pieces I know of (who knows what we haven't found yet). I've reinstalled the patch and checked 10x that the builds matched all around. They do. I've re-started S2 twice now. But the user experience and results are so bad I've now taken S2 off the load balancer and am directing all traffic only to S1. This is OK for a while but not what we need.

    But I'm sure without any explanation or apparent fix.

    Any ideas?
    Last edited by Steve Workings; 01-03-2018 at 04:28 PM.
    -Steve


  2. #2
    Member
    Real Name
    Sean OKelly
    Join Date
    Sep 2015
    Location
    Charleston, WV
    Posts
    117

    Default Re: Server problems

    Hi Steve,

    I'm wondering if the second server had a Windows Update processed that wasn't processed on the first server? If there was maybe an update to the database driver it could cause that kind of error. I've witnessed this in a corporate environment before but not with Alpha. Just a thought.

  3. #3
    Moderator
    Real Name
    Steve Workings
    Join Date
    Apr 2000
    Location
    The Dreaded Chair
    Posts
    5,516

    Default Re: Server problems

    Interesting thought. I compared the WindowsUpdate.log on the 2 servers. They're dis-similar enough I really don't know what to compare. But I see that a couple hours after my restart on Jan. 3 some potentially worrying messages:

    2018-01-03 15:27:50:495 940 918 Agent WARNING: failed to access the auth cab, fatal error 0x80070003
    2018-01-03 15:27:50:604 940 918 Agent WARNING: Invalid service in the backup data store; cleaning up
    2018-01-03 15:27:50:604 940 918 Agent WARNING: Failed to add and register service 117cab2d-82b1-4b5a-a08c-4d62dbee7782 to the data store 0x80240031
    2018-01-03 15:27:50:604 940 918 Agent WARNING: Service Recovery: Attempting to add pending registration for service 117cab2d-82b1-4b5a-a08c-4d62dbee7782 to the data store
    2018-01-03 15:27:50:604 940 918 SLS Retrieving SLS response from server using ETAG "4V8nqvoeoxgpu+6kKNNNGpLr4BCvPTmaz82CIDm5o5g=_1440"...
    2018-01-03 15:27:50:604 940 918 SLS Making request with URL HTTPS://sls.update.microsoft.com/SLS/{9482F4B4-E343-43B6-B170-9A65BC822C77}/x64/6.3.9600.0/0?CH=196&L=en-US&P=&PT=0x7&WUA=7.9.9600.18838
    2018-01-03 15:27:50:635 940 918 EP Got 9482F4B4-E343-43B6-B170-9A65BC822C77 redir SecondaryServiceAuth URL: "117cab2d-82b1-4b5a-a08c-4d62dbee7782"
    2018-01-03 15:27:50:698 940 918 SLS FATAL: SLS:CSLSRequest::RetrieveAdditionalAttributesIfRequired: CoCreateInstance failed with 0x80040154.
    2018-01-03 15:27:50:698 940 918 Agent WARNING: Failed to retrieve SLS response data for service 117cab2d-82b1-4b5a-a08c-4d62dbee7782, error = 0x80040154
    2018-01-03 15:27:50:698 940 918 Agent FATAL: Caller Service Recovery failed to opt in to service 117cab2d-82b1-4b5a-a08c-4d62dbee7782, hr=0X80040154
    2018-01-03 15:27:50:698 940 918 EP Got WSUS Client/Server URL: "http://wsussng0101.service.softlayer.com/ClientWebService/client.asmx"
    2018-01-03 15:27:50:698 940 918 Setup Checking for agent SelfUpdate
    2018-01-03 15:27:50:698 940 918 Setup Client version: Core: 7.9.9600.18838 Aux: 7.9.9600.18696
    2018-01-03 15:27:50:698 940 918 EP Got WSUS SelfUpdate URL: "http://wsussng0101.service.softlayer.com/selfupdate"
    2018-01-03 15:27:50:698 940 918 Misc Validating signature for C:\Windows\SoftwareDistribution\SelfUpdate\wuident.cab with dwProvFlags 0x00000080:
    2018-01-03 15:27:50:698 940 918 Misc Microsoft signed: NA
    2018-01-03 15:27:50:698 940 918 Misc Infrastructure signed: Yes
    2018-01-03 15:27:50:698 940 918 Misc Validating signature for C:\Windows\SoftwareDistribution\SelfUpdate\TMP2997.tmp with dwProvFlags 0x00000080:
    2018-01-03 15:27:50:714 940 918 Misc Microsoft signed: NA
    2018-01-03 15:27:50:714 940 918 Misc Infrastructure signed: Yes
    2018-01-03 15:27:50:776 940 918 Setup FATAL: GetClientUpdateUrl failed, err = 0x8024D009
    There's enough wrong with the way A5 behaves on S2 to strongly suggest something external to A5 and this could be it.

    What would be the next step to insuring that the Windows installation is correct?
    -Steve


  4. #4
    Moderator
    Real Name
    Steve Workings
    Join Date
    Apr 2000
    Location
    The Dreaded Chair
    Posts
    5,516

    Default Re: Server problems

    Found something more useful I think - the list of Updates installed.

    S1 was last updated on 10/1/2017 and has 5 important updates waiting.

    S2 had 3 security updates installed on 12/28/2017, about the time these problems started.

    What might I do about this? I definitely don't want to update S1 as it's the only reliable server of the 2 right now.
    -Steve


  5. #5
    "Certified" Alphaholic mikeallenbrown's Avatar
    Real Name
    Mike Brown
    Join Date
    Nov 2009
    Location
    United States
    Posts
    1,529

    Default Re: Server problems

    What about a system restore to a point prior to the bad updates?
    Mike Brown - Contact Me
    Programmatic Technologies, LLC
    Programmatic-Technologies.com
    Independent Developer & Consultant

  6. #6
    Moderator
    Real Name
    Steve Workings
    Join Date
    Apr 2000
    Location
    The Dreaded Chair
    Posts
    5,516

    Default Re: Server problems

    This is Windows Server 2012. I'm pretty tempted to uninstall the 3 updates made on 12/28 on S2. But I'm not very informed about this kind of thing - is it OK to do it, and then later on if I discover that was the cause, pick & choose what gets installed again? Given the worlds' (and my customers') security concerns, I don't know if I should be fooling with security updates. OTOH, if that's the cause and it puts me back in business, it may well be just fine.

    Or, not the problem at all but I don't have any other candidates, plus the timing on these installs is right in line with the beginning of the troubles.
    -Steve


  7. #7
    Moderator
    Real Name
    Steve Workings
    Join Date
    Apr 2000
    Location
    The Dreaded Chair
    Posts
    5,516

    Default Re: Server problems

    Update - One at a time, I uninstalled the updates of 12/28 and rebooted the server to see if any of them made the difference. None of them did -all the same problems still exist. It sure sounded like and still sounds like a very logical culprit, but that didn't do the job.
    -Steve


  8. #8
    Member
    Real Name
    Garth McConnochie
    Join Date
    May 2009
    Location
    UK
    Posts
    134

    Default Re: Server problems

    Happy New Year Steve...

    Is there any way you can clone the good server.... We've found that backing out windows updates can leave 'residue'

  9. #9
    Member
    Real Name
    Alex
    Join Date
    Oct 2015
    Location
    Perth, WA Australia
    Posts
    101

    Default Re: Server problems

    Steve i would be getting another server instance if possible. Sounds like something out of alpha servers control. I know AWS lets you clone your instances but by memory i believe you are not deploying to AWS.
    Alex Collier

    "Give a man a fish and you will feed him for a day; teach a man to fish and he will eat forever."

    AA Build 5213-5149 >> Deploying to IIS 8.5 on Windows 2012 R2 Server in AWS

  10. #10
    Member
    Real Name
    Rich Fulham
    Join Date
    May 2011
    Location
    Portland, OR
    Posts
    835

    Default Re: Server problems

    Steve,
    The updates rolled out by Microsoft or possibly anti-virus venders in December 2017 have left a number of our desktop application users in a bind. Maybe servers are having similar issues, so I will explain:

    Our desktop programs are not coded with Alpha Software, but the issues we are seeing could be relevant to any Windows applications. The recent Windows and/or anti-virus updates block unsigned DLL's (and I assume EXE's) from running under certain situations. We have seen where it matters how or where a shortcut is located that was used to launch the app.

    The DLL's from Alpha, that I have checked, under build 4642 were signed with a code certificate that expires January 20, 2018. I am wondering if there is a DLL or EXE that AA is calling which has not been code signed. Or, maybe the code certificate has expired or is too close to expiring.

    I am curious about the "OpenSSL" files used by AA. Maybe they have unsigned or expired certificates.

  11. #11
    Member
    Real Name
    Sean OKelly
    Join Date
    Sep 2015
    Location
    Charleston, WV
    Posts
    117

    Default Re: Server problems

    Hi Steve,

    One way or another the variation is in the server. If s2 is only used for this application I would simply recreate the server entirely and move forward. My guess is that recreating it will solve the issue and you won't need to spend a ton of time figuring out the issue. You could replicate the server first, thereby keeping a bad copy of it for debuging, end then recreate from scratch. Get the system up and running first, then go back and debug if you feel it's time well spent.

  12. #12
    Moderator
    Real Name
    Steve Workings
    Join Date
    Apr 2000
    Location
    The Dreaded Chair
    Posts
    5,516

    Default Re: Server problems

    I'm leaning in that direction Sean. Nobody's come up with an answer, and I sure have asked around.
    Thanks.
    -Steve


  13. #13
    Moderator
    Real Name
    Steve Workings
    Join Date
    Apr 2000
    Location
    The Dreaded Chair
    Posts
    5,516

    Default Re: Server problems

    Fixed, I think. Before asking Zebrahost for a server rebuild (I needed to keep the IP addresses and other complexities) I decided to copy the good S1 webroot over the bad S2 webroot (again, a little move complex than this but in the end, that's the gist of it). I didn't have much expectation because republishing to S2 wasn't making a whit of difference. But, I copied, then restarted the S2 server instances, ran tests on a few known problems and all seem resolved.

    We'll put S2 back into the load balancer rotation tomorrow mid-day and I'll update here if my assessment is wrong.
    -Steve


  14. #14
    "Certified" Alphaholic MoGrace's Avatar
    Real Name
    Robin
    Join Date
    Mar 2006
    Location
    Los Angeles
    Posts
    3,408

    Default Re: Server problems

    Wouldn't it be nice if MS would tell you what an update is scheduled to do BEFORE it does it? Like an actual release note with something better than the generic 'Security Update'...
    Robin

    Discernment is not needed in things that differ, but in those things that appear to be the same. - Miles Sanford

  15. #15
    Moderator
    Real Name
    Steve Workings
    Join Date
    Apr 2000
    Location
    The Dreaded Chair
    Posts
    5,516

    Default Re: Server problems

    It's really hard for me to tell what happened. I do know that in the last 24 hours I've been publishing my AEX to S1 and I cannot get it to effect any changes - even rebooted the machine in hopes of clearing the problem. So I still have "a problem" but have yet to really identify it. Among the puzzles is why republishing to S2 didn't make a difference, and now I'm having said problem publishing -- at least the AEX -- to S1.

    I've not made any changes in A5 since the November patch. Windows, of course, is constantly evolving which leads me to suspect a Windows/A5 clash. And I'm supposed to be developing product, not doing this.
    -Steve


  16. #16
    Moderator
    Real Name
    Steve Workings
    Join Date
    Apr 2000
    Location
    The Dreaded Chair
    Posts
    5,516

    Default Re: Server problems

    Update - yeah, I've proved that I cannot update the AEX on S1, at least. S1 has both development/publishing installation, and an A5 instance of the classic server. For two days now I've been trying to update a couple functions and publish an updated AEX to be used, but it's not having it. I can see the new date/time on the AEX in the webroot, but none of the changes are taking effect, including some simple save_to_file trace text to see if anything's happening. I've gone as far as re-booting the server, and no joy. I'll bet this publishing problem is/was related to the rest of the problems below and why I couldn't just publish my way out of the situation.

    So, what does one do when the AEX won't update?
    -Steve


  17. #17
    "Certified" Alphaholic
    Real Name
    Frank Bicknell
    Join Date
    Feb 2010
    Location
    39.1915° N, 84.4348° W
    Posts
    2,441

    Default Re: Server problems

    I recall having a similar situation a while back. When I published it looked as though it was successful but wasn't. The only way I could get the updates to work was to delete all of the files in the web root then republish.
    Win 10 64 Development, Win 7 64 WAS 11-1, 2, Win 10 64 AA-1,2, MySql, dbForge Studio The Best MySQL GUI Tool IMHO. http://www.devart.com/dbforge/mysql/studio/

  18. #18
    Member
    Real Name
    Rich Fulham
    Join Date
    May 2011
    Location
    Portland, OR
    Posts
    835

    Default Re: Server problems

    Steve,
    Given the latest information you posted, my guess would be that Windows Prefetch area needs to be cleaned out.

    https://helpdeskgeek.com/help-desk/d...dows-prefetch/

  19. #19
    Moderator
    Real Name
    Steve Workings
    Join Date
    Apr 2000
    Location
    The Dreaded Chair
    Posts
    5,516

    Default Re: Server problems

    Well, I tried a version of Frank's suggestion. Live server so I had to move quickly and small: I renamed the existing .aex so I'd have it, and quickly created a new one, then further cleared the server aex cache. No joy.

    And Rich, it appears from what I can tell the prefetch isn't enabled on Windows servers - this is a Windows 2012 server but the suggestion/solution would be consistent with the problem.
    -Steve


  20. #20
    Member
    Real Name
    Rich Fulham
    Join Date
    May 2011
    Location
    Portland, OR
    Posts
    835

    Default Re: Server problems

    Steve,
    Have you checked for multiple folder locations containing your .aex file on the server? What about the ProjectProperties.projectProperties file, does it contain a path to the .aex file? On my server it contains just the aex file name, no path. I see WAS has two configuration items dealing with AEX in the instance.xml file that A5Server is started with: 1) AEXInMemory.Active, and 2) AEXInMemory.MaxSize.

  21. #21
    Moderator
    Real Name
    Steve Workings
    Join Date
    Apr 2000
    Location
    The Dreaded Chair
    Posts
    5,516

    Default Re: Server problems

    Don't see any problems. It's interesting - I can update components and A5W pages just fine (did both today). But getting the server to see a new AEX and it's changes? Not happening.
    -Steve


  22. #22
    "Certified" Alphaholic
    Real Name
    Frank Bicknell
    Join Date
    Feb 2010
    Location
    39.1915° N, 84.4348° W
    Posts
    2,441

    Default Re: Server problems

    Can you examine the contents of the aex file on the server to see if it has the changes? I have never tried opened an aex file. Maybe you can compare the two files with a utility that does that. If it matches then it really seems like it has to be an Alpha issue.
    Another thought. Can you map the webroot and copy and paste the file to the server.
    Win 10 64 Development, Win 7 64 WAS 11-1, 2, Win 10 64 AA-1,2, MySql, dbForge Studio The Best MySQL GUI Tool IMHO. http://www.devart.com/dbforge/mysql/studio/

  23. #23
    Moderator
    Real Name
    Steve Workings
    Join Date
    Apr 2000
    Location
    The Dreaded Chair
    Posts
    5,516

    Default Re: Server problems

    All functions in the aex are encrypted - not sure anything would come of this.

    I'm not sure what you mean or what might be achieved with "map the webroot and copy and paste the file to the server."

    Thanks
    -Steve


  24. #24
    "Certified" Alphaholic CharlesParker's Avatar
    Real Name
    Charles Parker
    Join Date
    Dec 2012
    Location
    New Orleans, LA
    Posts
    1,871

    Default Re: Server problems

    Something seems odd here with this AEX file. Maybe this will help:
    https://www.alphasoftware.com/docume...pplication.xml
    NWCOPRO Nuisance Wildlife Control Software-My Application: http://www.nwcopro.com "I am not discouraged, because every wrong attempt discarded is another step forward."

  25. #25
    Moderator
    Real Name
    Steve Workings
    Join Date
    Apr 2000
    Location
    The Dreaded Chair
    Posts
    5,516

    Default Re: Server problems

    That's a decent primer on the AEX Charles. Alpha will tell you the server now recognizes any updated AEX without clearing the cache but my experience is different. Nothing here though I don't know and haven't tried.
    -Steve


  26. #26
    "Certified" Alphaholic
    Real Name
    Frank Bicknell
    Join Date
    Feb 2010
    Location
    39.1915° N, 84.4348° W
    Posts
    2,441

    Default Re: Server problems

    Quote Originally Posted by Steve Workings View Post
    I'm not sure what you mean or what might be achieved with "map the webroot and copy and paste the file to the server."
    Thanks
    I have my webroot on my server mapped to a drive on my desktop. I can copy and paste files just like it was a local drive. I can do it also from a remote location with RDP.
    I had a time when I published files and the publish process completed OK but the files on the server were not reflecting the changes. I couldn't figure out what the problem was i thought perhaps it was a cache issue. I started examining the files and found out that only the date was changing not the size even though the newly published files were larger. I finally deleted all of the files in the webroot on the server and republished and the problem went away and has never happened again.
    Win 10 64 Development, Win 7 64 WAS 11-1, 2, Win 10 64 AA-1,2, MySql, dbForge Studio The Best MySQL GUI Tool IMHO. http://www.devart.com/dbforge/mysql/studio/

  27. #27
    Moderator
    Real Name
    Steve Workings
    Join Date
    Apr 2000
    Location
    The Dreaded Chair
    Posts
    5,516

    Default Re: Server problems

    Well, I was one official build behind so I took a chance and upgraded to the current build. Now the servers recognize the AEX changes. I still don't know why the trouble, but it's over, thought I'd put a period on this one.

    So I'd say if this happens again (to me or anyone) start with a (re)install of the server patch. And if you have a good copy of the webroot, use that to replace your current webroot - sort of what Frank is suggesting that something is/was amiss there.

    Thanks for all input and time.
    -Steve


Similar Threads

  1. Activate web server license problems
    By Garry Flanigan in forum Application Server Version 11 - Web/Browser Applications
    Replies: 4
    Last Post: 09-12-2017, 07:30 PM
  2. v12 application server problems
    By Flommer in forum Mobile & Browser Applications
    Replies: 2
    Last Post: 11-06-2015, 01:15 PM
  3. Problems getting UDF recognised on the web server
    By Malcolm in forum Application Server Version 11 - Web/Browser Applications
    Replies: 2
    Last Post: 07-11-2012, 07:20 PM
  4. Problems publishing app to web server
    By envisionsolutions in forum Application Server Version 10 - Web/Browser Applications
    Replies: 3
    Last Post: 05-07-2010, 06:11 AM
  5. Network Server Problems
    By andrewsim in forum Alpha Five Version 4
    Replies: 6
    Last Post: 12-24-2002, 08:24 AM

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •