Monday, April 2, 2018

Planned web service outage on 4/2/18 is complete

The planned SAS web services outage has ended and all services have been restored. Please contact your LSP if you see any further problems.

Planned web services outage, 8:30 pm on 4/2/18


This evening starting at 8:30 pm all SAS web services will be down for a reboot of some critical infrastructure. We expect this outage will be relatively short, lasting only 30-60 minutes. We will update this page on our progress.

Friday, March 30, 2018

3/30/18, 3:50PM: All web services have been restored

3/30/18, 3:50PM: All services have been restored and we are monitoring the systems in case of recurrence.

3/30/18, 2:30PM: A storage issue has interrupted service for some SAS web sites.  We are working to restore service and hope to have all sites restored by 5pm.

Wednesday, March 28, 2018

Web -- Drupal updates complete, 8:20 pm


Emergency maintenance on all SAS Drupal websites is complete. Please notify your LSP if  you experience any problems.

Web -- Drupal update emergency maintenance status, 5:40 pm


A large proportion of SAS websites have been restored. These include all the sites hosted on our standard D6 and D7 webservers. We are continuing to work on websites that are hosted elsewhere. We will continue to announce restorations of service as they occur. 

Web -- Drupal update emergency maintenance status, 3:40 pm


All SAS websites (hosted on Drupal) are now offline for emergency maintenance. We will update this page as we make progress.


Web -- Drupal Emergency maintenance update status, 2:40 pm


The following sites are unavailable due to  emergency maintenance. Users will see a white screen that says "Pantheon," followed by a request for a password. They will not be able to log in.

Once the sites are restored, they will no longer be locked by password.

The sites are:

rees.sas.upenn.edu       
sals.sas.upenn.edu       
www.nettercenter.upenn.edu
pod.sas.upenn.edu       
omnia.sas.upenn.edu
amcs.sas.upenn.edu       
www.fels.upenn.edu     
isp.sas.upenn.edu       
cni.sas.upenn.edu       


Friday, March 23, 2018

Potential Scheduled Downtime for most SAS Web Services 3/28 2pm

Beginning at 2pm on Wednesday March 28th, most SAS web services may be offline for a critical security patch.  We are awaiting details from our vendor, but wanted to provide early notice of this potential outage.

If it is necessary, SAS web services will be taken offline as early as 2pm on 3/28 so that this security patch can be applied and tested.

We will reactivate each web server as soon as we can confirm it is secure and hope to have all services restored by early evening. 

Please monitor http://notify.sas.upenn.edu for updates as it will be unaffected by this outage.

Monday, March 12, 2018

Emergency maintenance is complete


The emergency maintenance that began at 5 pm today is complete, and all services have been restored.

If you should find a problem that we did not fix, please inform your lsp.


3/12/2018, 5PM: Emergency maintenance will affect many SAS web and departmental servers

3/12/2018, 5PM: In order to repair hardware that has failed, SAS Computing will conduct emergency maintenance beginning at 5pm.  During this maintenance many web and departmental services will be unavailable.  This will include:
  • Most websites hosted on SAS servers
  • web-facstaff
  • students.sas
  • dhcp services
  • writing.upenn.edu
  • groups.sas.upenn.edu mailing list service
  • hans.math.upenn.edu
Sending and receiving email should not be impacted except for groups.sas mailing lists and users who receiving mail @writing.upenn.edu.

We hope to restore most services by 9pm, and will provide updates here at http://notify.sas.upenn.edu.

Friday, March 9, 2018

3/12/2018: Groups.sas mailing list services currently unavailable

3/12/2018, 2PM: SASC staff have determined that the fix for groups.sas will require downtime for a large number of other services.  To avoid interruption of many services during working hours, we will be scheduling this maintenance for late this evening or early tomorrow morning.  We will update here with the planned outage window once it is set.

3/12/2018, 9:45AM: SASC staff continue to work to fix the underlying issues to restore groups.sas mailing list services.  We will provide another update no later than 2pm.

3/9/2018, 3PM: The groups.sas mailing list service is currently unavailable.  Mail sent to addresses @groups.sas.upenn.edu will be held and delivered once the service is restored.  We are working to identify the issue and restore service.

Friday, January 26, 2018

1/26/2018 Service restored; residual delays


The SAS email delivery servers are back up and running. Users may see delays in delivery this morning as the servers work through the backlog created by last night's outages.

1/26/2018, Intermittent problems with SAS mail delivery to O365 and google@sas

The SAS mail delivery servers are experiencing problems which have resulted in intermittent mail delivery delays to SAS O365 and google@sas email accounts. We are working on the problem and will update later this morning. Sending mail outbound should not be affected.

Thursday, October 26, 2017

10/26/2017 All web services restored


As of 12:50, all SAS web services and email delivery services have been restored. Please contact your LSP if you  find a service that's not working correctly.

10/26/17 - Outage of SAS web services

10:30AM: Major services have been restored.  We expect all services to be restored by 1pm. Next update at 1pm.


9AM: We have replaced the failed hardware and are beginning to restore services.  We will provide another update at 10:30AM.



7:30AM: As of early this morning we are experiencing an internal network/storage issue in our central servers.  As a result most SAS-hosted websites are currently unavailable.  Delivery of email should continue to work but delays in delivery are to be expected.  On-call staff are responding.  We will provide an update at 9am.

Tuesday, June 27, 2017

6/26: All services restored

The GPC research cluster has now been restored.

Saturday, June 24, 2017

6/23: Update after power outage

Groups.sas mailing list services have now been restored and delayed mailing list mail has been delivered.

We have temporarily disabled access to the GPC research cluster while we perform validation of the storage system in response to follow-on problems after the outage.  We expect to restore access to the GPC on Monday 6/26.

Wednesday, June 21, 2017

6/21/2017 7:30 PM Major Campus Poweroutage:

As of 7:30PM we believe almost all services have been restored, except for mailing list services offered by groups.sas and the GPC research cluster.

As of 4:30PM email services have been restored as have most web services.  We are continuing to restore some smaller, individual servers and check all services for continuing problems.  We will provide another update at 6PM.

As of 3:30PM incoming mail is starting to be delivered.  No mail should have been lost.  Mail from earlier in the day will be delivered as sending servers hand it off to our mail servers.  We are working to restore web and mailing list services next.  We will provide another update at 4:30PM.

As of 2:30PM we are still working to restore the central servers.  We are working through damage caused during the outage and hope to begin restoring services as soon as possible.  We will provide another update at 3:30PM.

As of 1 PM power has been restored to University Data Center. We are beginning the process of inspecting our servers and restoring services.  Windows file shares (U and S drives) have been restored.  We will provide another update at 2:30PM.

As of 12:30PM power has been restored to most buildings, but the central University Data Center is experiencing follow-on complications in their power equipment.  Once power is restored to the Data Center., SAS will begin restoring services.  At this time we do not have an estimate for restoration of these services.  Research cluster services hosted in the Franklin Building Annex have been restored.  We will provide another update at 2PM.

As of 11AM Power restoration is underway.  Buildings are regaining power and we are beginning the work of restoring power to the datacenter and our systems.  We hope to see some critical services restored by 1PM.  We will provide another update at 12:30PM. 

As of 9:45AM Upenn Facilities is estimating a restoration of power by approximately 12PM.  At that time we will begin restoring services.  At this time email routing, web services, and windows file servers are unavailable.  Other campus resources are impacted as well.  We will provide another update at 11:30AM.

At 8AM on 6/21/2017 Walnut street between 36th and 34th appears to have lost power. This has required a shutdown of most SAS server services.  Currently mail routing, web services, and some file server services are offline.  We will provide an update at 9:30AM.

6/21/2017 8AM: SAS Web Services Outage

As of 8AM on 6/21/2017 recovery is proceeding and we still hope to have all services restored by 9am.


On 6/21/2017 around 1AM, many SAS web services became unavailable due to hardware issues with one component of our web services infrastructure.  SAS Computing staff are responding and we hope to have all sites restored before 9AM.  We will post another update here at 8AM.

We apologize for this outage.

Monday, April 3, 2017

SAS Service Outages 4/3/2017 4:40PM

Beginning around 4:40PM on 4/3/2017 we experienced a significant network outage that interrupted many SAS email and web services. 

Services were brought back online individually with many services back by 6:30PM.  As of 10PM we believe all major services are restored.  If you are still seeing problems, please contact your lsp.

We apologize for this service interruption.

Sunday, August 7, 2016

Planned outage for 8/7/16 is complete.

Scheduled network maintenance is complete. All services are operating normally at this time.

Thursday, July 28, 2016

Planned Network Outage Sunday, August 7, 8-8:30 am

On Sunday, August 7, there will be a planned network outage in one of the primary SAS data centers. ISC predicts the outage will last around 10 minutes but reserves 30 minutes between 8:00 - 8:30 am that day.

A wide variety of services will be unavailable during this networking downtime:

* Routing of mail to O365 email and calendar, and google@sas. (All mail will be delivered once networking is restored.)

* SAS Web Services (including www.sas.upenn.edu and most other sites hosted on central SAS servers)

* SAS Central File Servers (home directories and shared drives for many departments)

* SAS Print Server (network printers for some departments)

* SAS Media Server (audio and video recordings)

* Web and VPN Remote Access Server (remote.sas)

Please watch http://notify.sas.upenn.edu/ for additional details and updates.

If you have questions or concerns about this outage, please contact your Local Support Provider.

Thursday, June 30, 2016

Email Delivery delays into Office365

We are seeing significant delays in email coming in to O365.  Microsoft has confirmed that this is an issue affecting many O365 users, not just Penn and they are treating it as their highest priority issue.  No email is being lost.  All email will be delivered once the issue is resolved.

1:30PM: Microsoft has identified the issue and is working on resolution now.

2:30PM: More mail is getting through, but significant delays still exist.  Microsoft is making progress on restoring the full service.

3:30PM: Mail continues to be processed slowly.  Microsoft indicates that they are working through the backlog of email.

4:30PM: Microsoft continues to process the backlog and is implementing a fix to speed up the process.

5:45PM: Service is returning to normal and mail from earlier today is now being processed more quickly.  Please note that depending on your mail client configuration you may need to scroll down to see mail from earlier in the day that is now being delivered.

7PM: All email from earlier in the day has now been delivered.  All new mail is being delivered as it arrives.  We apologize for the outage.

Sunday, August 2, 2015

Preventive maintenance complete and services restored

8/2/15, 2 pm
All ISUS systems, including mail, web services, departmental servers, and Oracle application servers, have been restored.

Still working on restoring services

8/2/12:40 pm
Email services on facstaff accounts have been restored. Routing to google@sas accounts is working. We continue to work on restoring other services such as departmental servers and we will provide an update again this afternoon.

8/2/12:10 pm
All SAS websites managed by ISUS have been restored. Most Oracle applications are available. We are continuing to work on all other services and will provide an update again this afternoon. We apologize for the inconvenience.

8/2/15 10:10 am.
Computing staff are onsite and still working to restore services.We apologize for the continued outage. We will update again by 12 noon at the latest.

Saturday, August 1, 2015

8/1/15 9:10 pm.
 Facstaff webmail services were unexpectedly interrupted for a short while this evening, but have been restored until 10 PM.

Websites had to be taken down earlier than originally scheduled, and will not be available until restart tomorrow morning. We apologize for the inconvenience.

Wednesday, July 1, 2015

Scheduled Power Outage, August 1st, 2015

On Saturday, August 1, 2015 from 10PM - Sunday 10AM, critical maintenance will be performed on electrical systems in the central University data center.

During this time, many SAS IT Services will be unavailable.

The following SAS services will be unavailable during this power outage:

* SAS Email (Facstaff, Webmail, and mail forwarding to Google@SAS and all external email providers)

* SAS Web Services (including www.sas.upenn.edu and most other site hosted on central SAS servers)

* SAS Central File Servers (home directories and shared drives for many departments)

* SAS Calendar Services (Meeting Maker)

* SAS Print Server (network printers for some departments)

* SAS Media Server (audio and video recordings)

* Web and VPN Remote Access Server (remote.sas)

For information about other campus IT systems impacted by this scheduled maintenance, please see: https://www.isc.upenn.edu/alerts-outages/planned-3401-walnut-data-center-outage-812015-822015

Please watch http://notify.sas.upenn.edu/ for additional details and updates.

If you have questions or concerns about this outage, please contact your Local Support Provider.

Saturday, February 28, 2015

2/28/15 4:15PM: Restarting Server

7:33PM: All services should now be restored.

5:33PM: Systems are coming back online now.  We expect most systems to be live by 7PM.

4:15PM: In order to finalize repairs after the outage this morning we are restarting a failed server.  Various web services may be unavailable over the next one hour.  We will provide an update by 5:30pm.

2/28 1AM - 5AM Facstaff Outage

5 AM Facstaff email service is restored.
3:15AM: On-call staff are on-site and working to restore service.  We hope to have service restored by 5AM.  We will provide an update no later than 5AM.

2:10AM: We have identified the problem and on-call staff are en route.  At this time we believe the outage is restricted to facstaff mail, and all web systems are currently available.  We will provide another update by 3:15AM

1AM: We are experiencing hardware problems with one of our servers.  This has interrupted connections to some of our email and web services.  We are currently diagnosing the problem and will rpovide and update by 2:15AM.

Monday, October 13, 2014

Services restored 10/13 1:30pm

All services were restored at 1:30pm.  We apologize for the outage.

Networking issue in campus server room

We are experiencing a broad networking outage in a central server room on campus.  We are seeing problems connecting to many SAS systems as of 1pm Monday 10/13.  We will provide an update by 1:30pm

Sunday, October 5, 2014

Scheduled Outage Completed at 5am

ISC completed the networking maintenance early and SAS systems became available again shortly after 5am.

If you are still encountering any problems reaching SAS systems, please let your LSP know.

Thanks!

Monday, September 29, 2014

Scheduled Network Outage, 4AM - 6AM, Sunday October 5th


On Sunday, October 5th, 2014 from 4 AM - 6 AM, critical maintenance will be performed on networking systems in our data center.  During this time, many SAS IT Services will be unavailable.

The following SAS services will be unavailable during this outage:

* SAS Email (Facstaff, Webmail, and mail forwarding to Google@SAS and all external email providers)

* SAS Web Services (including www.sas.upenn.edu and most other site hosted on central SAS servers)

* SAS Central File Servers (home directories and shared drives for many departments)

* SAS Calendar Services (Meeting Maker)

* SAS Print Server (network printers for some departments)

* SAS Media Server (audio and video recordings)

* Web and VPN Remote Access Server (remote.sas)

We will provide an update here as soon as work has been completed.

If you have questions or concerns about this outage, please contact your Local Support Provider.

Saturday, October 19, 2013

Campus network problems

We have a problem report for networking to upenn sites from off campus. This is confirmed from verizon and comcast users.

As of 10:11 Verizon and Comcast user are reporting restored service.
It appears service broke around 9:10 AM.

Sunday, February 10, 2013

Reports of Networking Problems to Campus

As of 5:40PM on 2/10 we are receiving reports of problems reaching various systems on campus from off-campus locations.  We can verify that some users are unable to connect to campus servers.  We have reports from both Comcast and Verizon wireless customers so far.

We are reporting this issue to the ISC NOC.

Thursday, January 17, 2013

1/17/13 10:30PM: Services Restored

All services are restored.

Behind the scenes we are moving systems off of the problematic storage system.  This may cause some slowness during the moves.

We hope that by 9AM no major systems will be impacted if the storage problem recurs.

1/17/13, 8:30PM: Services restored

All mail and most web services are now restored.  We are working to find a permanent resolution the storage instability.  We will provide another update at 10PM.

1/17/13: 7:45PM: Restoration continuing

We expect mail services to resume by 8:45PM.

Web services will follow. 

We apologize for the inconvenience.

1/17/13, 6PM: New Outage

I'm sorry to announce that the disk system has failed again.  We are working to bring services back and will provide an update at 7:30PM.

1/17/13: 10AM: All services restored

By 10AM all services were restored.

We apologize for the inconvenience.

1/17/13, 8:10AM: Most Services Restored

Most web and mail services are now restored.  Mail sent between 4AM and 8AM will be filtering in as it is processed.


1/17/13: 7:15AM: Update on Outages

We have identified the problem and are beginning to restore services.  We hope to have all services restored by 9am.

1/17/13: 5:30AM: Major Network Issue Impacting Many SAS Services

As of 5:30AM on 1/17/13, many SAS services are currently offline.  Including:

* Email routing for facstaff and students
* Most web services

We are working on diagnosing the problem and will have more details at 7AM.

Sunday, October 28, 2012

10/28:Hurricane Sandy

Please monitor: http://www.publicsafety.upenn.edu/ for updates on Penn's response to Hurricane Sandy.  Per that page:

----------------------
UPDATE OCTOBER 28, 2012 9:30AM:
CLASSES CANCELLED FOR MONDAY AND TUESDAY
Due to severe weather that is predicted to accompany the arrival of Hurricane Sandy, all classes at the University of Pennsylvania are cancelled for Monday, October 29 and Tuesday, October 30. Classes will resume with a normal schedule on Wednesday, October 31.
Only those personnel providing essential services should report to work on Monday and Tuesday.  An explanation for faculty and staff on the University’s policies regarding closures can be found at the following URL:
https://www.hr.upenn.edu/myhr/resources/policy/other/suspensionofnormaloperations
Penn’s Crisis Management Team continues to monitor the situation, and extensive preparations have been taken to ensure the safety of students and others who will be on campus during the storm.
Updates concerning the storm will be posted on the Penn homepage (www.upenn.edu) and the Public Safety web site (http://www.publicsafety.upenn.edu).  Faculty, staff and students can also call 215-898-MELT (215-898-6358) to hear a recorded message with any updates regarding the hurricane.
This is a major storm that will bring high winds and significant rain to the area. Localized flooding and some power outages are expected during the storm. We urge everyone to exercise extreme caution during this time. Information regarding hurricane preparedness can be found at the following link from FEMA: www.ready.gov/hurricanes

-------------------------

In the event of an outage of SAS technology systems we will update this site.  Please continue to check http://notify.sas.upenn.edu for updates.

Thursday, July 26, 2012

7/26: All SAS Services Now Restored

The network upgrade is complete and all SAS Services were restored before 7am.  If you notice any issues, please contact your LSP.

Sunday, July 22, 2012

Scheduled Network Outage 7/26/2012

On Thursday, July 26th 2012, from 6am-7am, critical maintenance will be performed on networking systems in our data center.

During this time, many SAS IT Services will be unavailable, including:

* SAS Email (Facstaff, Mail.sas, Webmail, and mail forwarding to Google@SAS and all external email providers)
* SAS Web Services (including www.sas.upenn.edu and most other sites hosted on central SAS servers)
* SAS Central File Servers (home directories and shared drives for many departments)
* SAS Calendar Services (Meeting Maker)
* SAS Print Server (network printers for some departments)
* SAS Media Server (audio and video recordings)

We expect most services to be restored well before 7am. 

Please watch this site for additional details and updates.

If you have questions or concerns about this outage, please contact your Local Support Provider.

Tuesday, May 15, 2012

Tuesday, November 8, 2011

COMPLETED: Scheduled Network Outage Tuesday, Nov 8, 5am-7am

As of 5:30AM all network services are available again.

Friday, November 4, 2011

Scheduled Network Outage Tuesday, Nov 8, 5am-7am

On Tuesday, November 8th, from 5am-7am, critical maintenance will be performed on networking systems in our data center.

During this time, many SAS IT Services will be unavailable.

The following SAS services will be unavailable during this outage:

* SAS Email (Facstaff, mail.sas, Webmail, and mail forwarding to Penn Live and all external email providers)
* SAS Web Services (including www.sas.upenn.edu and most other sites hosted on central SAS servers)
* SAS Central File Servers (Windows home directories and shared drives for many departments)
* media.sas.upenn.edu server (audio and video recordings)
* SAS Calendar Services (Meeting Maker)
* SAS Print Server (network printers for some departments)
* SAS FileMaker Database Server
* Footprints Server
* Portfolio Server
* Web and VPN Remote Access Server
* Symantec Endpoint Protection Management Server
* One Windows Active Directory Domain Controller, (other domain controllers will continue to be online, so domain logins will continue to be possible)

Sunday, September 18, 2011

RESOLVED: Networking Outage 9/18/2011 - 10:30PM

As of 11:10PM all services have been restored.



As of 10:30PM we are experiencing a major networking outage in the server room. www.sas.upenn.edu and facstaff email services are currently unavailable. Oncall staff are responding and we hope to restore service by midnight.

Saturday, May 28, 2011

Networking Outage 5/28/2011 - RESOLVED

As of 10:40PM these issues are now resolved.

We are experiencing a significant network outage as of 10PM 5/28/2011. Many SAS systems are currently unreachable including SAS email and web services. Networking staff are working on the problem now.

Tuesday, March 8, 2011

DRL power outage

There is a scheduled power outage tonight from 7PM to tomorrow 3AM.
Access interruption to the following departmental servers is expected during the outage:

http://www.ling.upenn.edu
http://www.math.upenn.edu
http://www.physics.upenn.edu

UPDATE: this outage proceded as expected; all services are now available.

Friday, January 21, 2011

RESOLVED: January 21, 2011 -- Facstaff Service Disruption

Friday, 1/21 -- 5:40pm

Service on facstaff.sas is restored

Friday, 1/21 -- 3:50pm

Facstaff, SAS' faculty/staff email server, is currently experiencing issues sending and receiving mail. Our mail server administrators are working on the problem now, and we expect the issue to be resolved within the hour. Please check back here for updates.

Thursday, November 11, 2010

Hans.math downtime 20101113 @11PM

The reboot scheduled for Friday, November 13, has been rescheduled to today at 11 PM.

Tuesday, July 13, 2010

6:45PM DRL Network Available

All network services have now been restored to DRL. Thank you for your patience and understanding.

DRL Network Outage

David Rittenhouse Labs is experiencing flooding that has disabled the building network equipment. All Pennnet based services are currently disabled. Staff are working to move the networking equipment to a new location.

Those departmental servers, that are affected by this outage, will be redirected here until the problem is resolved.

1PM Update: We expect network service to DRL to be restored by the end of the day.


Sunday, June 20, 2010

Try webmailfs2.sas.upenn.edu

If you are trying to reach webmailfs.sas.upenn.edu and your computer is still redirecting you to this page, please try going to https://webmailfs2.sas.upenn.edu .

This problem is a result of your web browser or computer caching the redirect. By using the temporary address https://webmailfs2.sas.upenn.edu you should be able to immediately reach our Webmail service.

Thanks.

Services Returning to Normal

SAS email and web services should now be restored fully. Mail sent during the outage will be delivered throughout the rest of the day. If you are experiencing any residual problems with SAS email or web services, please contact your LSP.

Monday, May 10, 2010

Scheduled Power Outage, June 20, 2010

On June 20, 2010 from 12:01 AM - 12 noon, critical maintenance will be performed on electrical systems in several buildings on campus. During this time, many SAS IT Services will be unavailable.

The following SAS services will be unavailable during this power outage:
  • SAS Email (Facstaff, mail.sas, Webmail, and mail forwarding to Penn Live and all external email providers)
  • SAS Web Services (including www.sas.upenn.edu and most other sites hosted on central SAS servers)
  • SAS Central File Servers (Windows home directories and shared drives for many departments)
  • media.sas.upenn.edu server (audio and video recordings)
  • SAS Calendar Services (Meeting Maker)
  • SAS Print Server (network printers for some departments)
  • SAS FileMaker Database Server
  • Footprints Server
  • Portfolio Server
  • Web and VPN Remote Access Server
  • Symantec Endpoint Protection Management Server
  • Windows Active Directory Domain Controller, jade.sas.upenn.edu (other domain controllers will continue to be online, so domain logins will continue to be possible)

For information about other campus IT systems impacted by this scheduled maintenance, please see: http://www.upenn.edu/computing/june-outage/

Please watch http://notify.sas.upenn.edu/ for additional details and updates.