Introduced a Glimmer component for managing pipeline secrets in the new UI to enhance user experience and functionality. See PR.
Introduced a new Glimmer component to manage API tokens, allowing users to create and manage them more efficiently. See PR.
PR Desync: a feature in the UI to detect and notify users of desynced pull requests in the database due to prolonged pipeline inactivity, providing a convenient resync button. See PR.
Support for reverting overridden secrets and improved ambiguity in secret deletion messaging in the new UI. See PR.
Introduced a new navigation bar for switching between secrets and tokens in the new UI. See PR.
Default status sorting added to the event jobs table, with sorting based on build color and then alphabetically by stage and job name. As depicted below, the failed build is sorted at the top. See PR.
API
Added feature to trigger jobs when a pull_request closed webhook event is received in Screwdriver. See PR and guide.
Added feature to set event status when a build is updated, tracking overall execution status for an event. See PR.
Added a DELETING state to the pipeline to prevent updates during pipeline deletion, addressing remaining pipeline data post-deletion issue. See PR.
Implemented a feature flag to control log payload logging, aiding log volume management. See PR.
Pipeline admin user list now updated based on SD user permissions table. See PR.
Bug Fixes
UI
Implemented a fix to allow navbar to be displayed when the hamburger menu is expanded on small devices. See PR.
Fixes issues with edit secret modal in new landing page by setting secret value properly for API request and returning full response when “allow in pr” field is changed. See PR.
Ensures that the workflow graph maintains the toggle state of downstream triggers when switching between different pipeline events. See PR.
Event rail reloading logic in the new pipeline landing page was adjusted to align with the vertical collection component. See PR.
Fixes an issue where cacheKey was incorrectly removed during data reloading in the pipeline workflow, ensuring the latest commit SHA is correctly reset. See PR.
Pipeline header updating issue resolved by adding necessary tracking to update properly when a pipeline changes. See PR.
Pipeline root directory now included and displayed alongside branch name in new UI. See PR.
API
Added a check for adminUserIds for backward compatibility in situations where old pipelines have a null adminUserIds column due to inability of MySQL 5 version to set default values in TEXT fields. See PR.
Fix for triggering the next build on the child event when some builds are restarted before all builds are completed. See PR.
Compatibility List
In order to have these improvements, you will need these minimum versions:
API - v8.0.26
UI - v1.0.1272
Store - v7.0.0
Queue-Service - v5.0.1
Launcher - v6.0.221
Build Cluster Worker - v5.0.1
Contributors
Thanks to the following contributors for making this feature possible.
Akinori
Keisuke
Ming
Pritam
Sagar
Vonny
Yuki
Yuta
Jithin
Questions and Suggestions
We’d love to hear from you. If you have any questions, please feel free to reach out here. You can also visit us on Github and Slack.
Author
Vonny Jap, Senior Manager, Software Dev Engineering, Yahoo
The Screwdriver team is pleased to announce our newest release which brings in new features and bug fixes across various components.
New Features
UI
The new Pipeline landing page is now generally available. Once users opt in, they will be redirected to the new experience.
The expand and collapse feature has been added to pipeline stages. The stage's completion status is now reflected in its border.
The display of multiple banners has been modified from a stacked format to a paginated layout
API
Relax permission check to allow admins from other SCMs to update pipeline
Added APIs to create and delete buildCluster annotations, enabling Screwdriver admins to move jobs to specific clusters
New endpoint to update buildcluster of a pipeline
Improve pipeline searches by adding the 'scmUri' query parameter
Introduced a 'not equal' filter for pipeline events creator in the pipeline events list API
Changes to banner to include scope and scopeId
Bug Fixes
UI
Pipeline header updating issue resolved by adding necessary tracking to the pipeline header component, allowing proper updates when a pipeline changes, see PR
Pipeline root directory now included and displayed with branch name in new landing page, with additional CSS styling cleanup, see PR
Corrected the aggressive logic in the new landing page that was preventing jobs included in a stage from being started/restarted, now allowing these jobs to be started/restarted as needed, see PR
Fixed missing stages border when accessing pipeline from Collection by directly requesting stage data from the backend, see PR
Enhanced UI consistency by applying standardized CSS styling to the modal button within the build detail page, aligning it with the appearance of the workflow graph tooltip start modal, see PR
Ensured the start all children pipelines API call requests the entire error object, allowing more detailed analysis of the API call and returned status code, see PR
Fix: Remote trigger's pipeline name now uses 'remoteName' instead of 'displayName' to avoid confusion. See PR
Fix: Build status determination was updated by relaxing the build meta warning object check and included warning statuses in the event card warning count. See PR
Fix: The edges from stages to upstreams are now properly drawn by providing a solution for insufficient horizontal spacing between stages. See PR
Fix: The post body for PR events was improved by correctly including the job prefix in the `startFrom` value when restarting a PR job. See PR
Fix: Column name in event job display changed from 'HISTORY' to 'STATUS' for accuracy in new landing page. See PR
Fix: Long branch names on the search page are now truncated. See PR
Fix: API fetch updated to ignore scheduler events for configured pipeline. See PR
Fix: The new landing page now correctly reloads the workflow graph when starting a new event. See PR
Fix: Overflow handling for long messages added to prevent text overflow in event card. See PR
Fix: Pipeline data removed from route models due to direct injection into components. See PR
Fix: The maximum node depth issue was fixed, addressing problems with 0-based counted node depth and empty arrays for pipelines with detached jobs only. See PR
Fix: The creation of a sparse array with empty slots which caused downstream calculations to break was fixed. See PR
Fix: Parameters drop down issue resolved in new landing page by adjusting dropdown spacing and removing additional scroll logic. See PR
Fix: Build warning determination logic was fixed to correctly display warning status. See PR
Fix: Node depth calculation in workflow graph was improved by introducing 'Topological sort' for more accurate depth assignments. See PR
Fix: An issue preventing the use of anchor tags within the navigation banner was fixed. See PR
Fix: Job restart is now disabled for jobs with no builds in the new landing page. See PR
API
Upgrade of Node.js v22 and dependencies
Addressed a bug in build triggering in certain cases of restarts, ensuring the next build is correctly triggered from the latest child event, see PR
Fixed an issue where an external trigger couldn't trigger a pipeline stage due to a misinterpretation of the event associated with the pipeline the job belongs to, see PR
Avoided creation of empty events when changed file names match the source directory, see PR
ParentBuildId now set just before execution to ensure correct metadata capture from parent builds, especially when they finish simultaneously, see PR
Converted the 'isActive' parameter from string to boolean for MySQL on banner list endpoint to ensure proper banner list retrieval, see PR
Included a webhook executor to the admin group before retrieving the pipeline token. This resolved the 'Pipeline has no admin' error during the webhook process, see PR
The metadata from the job that directly precedes the virtual job is now prioritized over the event metadata, see PR
Proper handling of virtual jobs in remote triggers, see PR
Fixed duplicate build issue by preventing the build status from updating to RUNNING if a build with the same ID is already running, see PR
Virtual builds now marked as completed when triggered by webhook, resolving a regression introduced in a previous update, see PR
Compatibility List
In order to have these improvements, you will need these minimum versions:
API - v8.0.16
UI - v1.0.1241
Store - v7.0.0
Queue-Service - v5.0.1
Launcher - v6.0.221
Build Cluster Worker - v5.0.1
Contributors
Thanks to the following contributors for making this feature possible.
Akinori
Keisuke
Ming
Pritam
Sagar
Vonny
Yuki
Yuta
Teppei Minegishi
Questions and Suggestions
We’d love to hear from you. If you have any questions, please feel free to reach out here. You can also visit us on Github and Slack.
Author
Vonny Jap, Senior Manager, Software Dev Engineering, Yahoo
We’re thankful for you! The Screwdriver team is pleased to announce our newest release which brings in new features and bug fixes across various components.
New Features
UI
Virtual job improvements - You can now see a different icon for virtual jobs in the workflow graph. They will also have a `virtual` tag in the pipeline Options tab. The virtual job build page will no longer show logs or warnings. You can read more about using `screwdriver.cd/virtualJob` annotation in the Screwdriver documentation.
Option to download directory artifacts as zipped file - You can now download an artifact directory for a build as a zipped file. Navigate to a folder under Artifacts and click the existing “Download” button. Max size 2GB.
Jobs list view has stages column and ability to select number of historical builds shown - You can now see which stage each job belongs to and sort by stage. You can also select to show up to 30 historical builds on the same page.
API
Add option to download directory artifacts as zip file - You can now download a directory of artifacts in a build as a zipped file using GET /builds/{id}/artifacts/this/is/a/directory/path?type=download&dir=true endpoint.
Bug Fixes
UI
Add feature flag for download all artifacts
Build detail view - Hide steps and logs for virtual builds
Download directory should use feature flag
Fix the missing download button
Find workflow graph node for a virtual build by job name
Show download button when the flag is false
Allow DOWNLOAD_ARTIFACT_DIR to be set in environment variables
API
Update pipeline-templates.md
Add the design implementation for the sd artifacts download
Set build status message while skipping execution of virtual job
Use query param dir=true to indicate directory download
Fix assigned remote trigger events
Add maxDownloadSize for artifact directory download
Fix the missing artifactsMaxDownloadSize
Improve metadata functional tests
Queue-service
Set build status message type while skipping execution of virtual job
Launcher
Add GIT_ASKPASS
ParentEvent should be a predecessor of other metadata
Event and parentEvent meta should be mutually exclusive
Downgrade to older version of sd-packages
Always allow parent event metadata to be merged during the build
Pin version of store-cli to v0.0.71
Compatibility List
In order to have these improvements, you will need these minimum versions:
API - v7.0.227
UI - v1.0.1095
Store - v6.0.0
Queue-Service - v4.0.5
Launcher - v6.0.210
Build Cluster Worker - v4.0.3
Contributors
Thanks to the following contributors for making this feature possible:
Akinori
Jithin
Keisuke
Ming
Pritam
Sagar
Tiffany
Vonny
Yuki
Yuta
Questions and Suggestions
We’d love to hear from you. If you have any questions, please feel free to reach out here. You can also visit us on Github and Slack.
Spoooky season is here! The Screwdriver team is pleased to announce our newest release which brings in new features and bug fixes across various components.
New Features
UI
Show remote trigger's displayName as title - You can now see the remote trigger’s pipeline name and branch by hovering on its node in the workflow graph.
Option to download all artifacts as zipped file - You can now download all artifacts for a build as a zipped file. Navigate to root artifact folder and you should see Download All button.
API
Add option to download all artifacts as zip file - You can now download all artifacts in a build as a zipped file using GET /v4/builds/{buildId}/artifacts endpoint.
Bug Fixes
UI
Bump data-schema to 24.0.0
Job name is omitted on the workflow graph when its displayName is shorter than its name
Bump elliptic from 6.5.4 to 6.5.7
Bump body-parser and express
API
Upgrade to latest command validator
Accelerate the functest
Run all scenarios in functest
Add stage join trigger case and stabilize stages test
Deep merge build meta into event meta
Create new event meta when deep merging build meta into event meta
Init a value when key does not exist in an object
Remove k8s-vm dependence
Enable sonar plugin config
Use another headless user x1 to run func tests against prod
Honor freeze windows for virtual jobs
Stabilize functional tests
Allow groupEventId filter for list pipeline events
Revert "fix(3055): remove k8s-vm dependence"
sd-setup-scm functest initialization
Store
Move pipeline template workflowGraph out of config into a new field
Queue-service
Skip execution of virtual job and mark it as SUCCESS
Move workflowGraph out of config and keep it at the same level as config
Launcher
Use new sd-package versions
Update Dockerfile with new sd-pkg versions
Internals
sd-packages: Publish aarch64 version for ztsd
Compatibility List
In order to have these improvements, you will need these minimum versions:
API - v7.0.211
UI - v1.0.1048
Store - v6.0.0
Queue-Service - v4.0.4
Launcher - v6.0.202
Build Cluster Worker - v4.0.2
Contributors
Thanks to the following contributors for making this feature possible:
Akinori
Ming
Pritam
Sagar
Tiffany
Vonny
Yuki
Questions and Suggestions
We’d love to hear from you. If you have any questions, please feel free to reach out here. You can also visit us on Github and Slack.
The end of summer is approaching! The Screwdriver team is pleased to announce our newest release which brings in new features and bug fixes across various components.
New Features
UI
Allow starting/restarting job from middle of stage - You can now start/restart the event from a job that is in the middle of a stage. Caveats: Stage setup will always rerun and you cannot rerun stage teardown directly.
Allow starting/restarting detached stage - You can also start/restart a detached stage. Click on the three dots on the top right corner of the stage to see the prompt.
API
Add endpoint to delete specific a command version - You can now remove a specific version of a command using DELETE /v4/commands/{namespace}/{name}/versions/{version} endpoint.
Add greater than/less than eventId filter for list pipeline events, sort and sortBy - You can now filter event IDs for a pipeline with greater than or less than filters. For example, GET /v4/pipelines/{pipelineId}/events?id=lt:{eventId}&count=5&sort=ascending or GET /v4/pipelines/{pipelineId}/events?id=gt:{eventId}&count=5
Bug Fixes
UI
Add cache, subscribe, and stages fields to pipeline template detail page
Bump ws and socket.io
Moved default workflow graph for pipeline template
Fix admin description
Make long build parameters confirmable
Unhide start and restart buttons for all stage jobs
Show all jobs in the job list view
Parameter bug for pipelines where only job level parameters exist
Fix job list view test
Fix job list view to load jobs sequentially
Show long parameter names as title
Fix some bugs with pipeline options page
Display parameter form when click start button
Build parameter tooltip style when a value is short or a description is defined
Long parameter is not nowrap
Move pipeline template workflowGraph out of config into a new field
Workflow graph is not rendering in pipeline template detail page
API
Do not allow start from stage teardown
Pass scmRepo in without making extra calls to GitHub
Add functional tests for sd-setup-scm step
Upgrade stream from v0.0.2 to v0.0.3
Add design document for pipeline templates
Skip execution of jobs between stage setup and event startFrom (only if they belong to the same stage)
Stabilize functional test
Stages metadata passed to stage teardown is incomplete when there is failure in stage job
Remote join will wait on downstream jobs during restart
Functional test improvements: Split beta tests into two parts, Run prod job after all beta done, Set the correct secrets and disable pipeline-template, Run tests in different jobs, Change the name of sd functional test x1, Enable pipeline template functional tests
Move pipeline template workflowGraph out of config into its own field
Internals
sd-packages: Use almalinux image
Compatibility List
In order to have these improvements, you will need these minimum versions:
API - v7.0.189
UI - v1.0.1012
Store - v5.2.0
Queue-Service - v3.0.10
Launcher - v6.0.200
Build Cluster Worker - v3.1.1
Contributors
Thanks to the following contributors for making this feature possible:
Aishwarya
Akinori
Hamza
Keisuke
Ming
Pritam
Sagar
Tiffany
Vonny
Yuki
Yuta
Questions and Suggestions
We’d love to hear from you. If you have any questions, please feel free to reach out here. You can also visit us on Github and Slack.
It’s summer! The Screwdriver team is pleased to announce our newest release which brings in new features and bug fixes across various components.
New Features
UI
Display admins on pipeline options page
API
Skip execution of virtual jobs - A job can be marked as virtual by adding the annotation screwdriver.cd/virtualJob: true. These jobs will not be executed while processing the workflow for an event, but will proceed with SUCCESS status. These jobs will skip queueing and execution and builds of its downstream jobs will be immediately created to continue the processing of the workflow.
Get repository data from scmRepo without GitHub access - Before this change, extra calls were made to GitHub to fetch SCM repository information. After this change, the API now will pass scmRepo info directly in order to avoid extra GitHub calls. This has been implemented for webhooks only so far.
Pipeline template enhancements: Can customize image, settings, environment, and requires for jobs that already exist in pipeline template, Add new jobs that are not part of pipeline template in user yaml, Flatten shared steps to jobs in pipeline template, Pipeline template validator shows extended configs for job template, Pipeline template validator shows workflow graph
Store
Replace jwt expiration with a config and environment variable - The jwt expiration is currently 13h and is written directly in the source code. This makes it difficult for SD administrators to change to arbitrary values to suit their use cases. Now Screwdriver cluster admins can set the JWT expiration length using JWT_MAX_AGE.
Add handling for the Expect header at onPreAuth - To reduce unnecessary retries when uploading large files, Screwdriver will use the Expect: 100-continue header. The content length can be checked before the body is uploaded. Thus, the HTTP client (store-cli) can receive a 413 error earlier without unnecessary retries.
Bug Fixes
UI
Show workflowGraph for pipeline template validation
Show missing fields in pipeline template detail page
Add test for parameter-utils
New V2 work: Use unified modal style for toggle job modal, Refactor parameter flattening to generalized util, Refactor graph rendering logic, Add v2 secrets route, New confirm action modal UI, case where there are no parameter groups to expand, Clean up the new v2 layout, Refactor pipeline secrets to not use ember data, Refactor to no use ember data for pipeline data, Screwdriver style mixin for Bootstrap buttons, add options to v2 route, Modal for adding a pipeline to a collection, new pipeline header component, Expand default modal styles, Update add to collection modal
API
Add logic to switch to using current user’s SCM token next
Fixed error handling of external trigger
Fix event where a restarted join build is run
Check original job is not archived
Handle API functional testing
Internals
Allow colons to be used in root directory
Allow to cd to rootDir containing special characters
Compatibility List
In order to have these improvements, you will need these minimum versions:
API - v7.0.162
UI - v1.0.967
Store - v5.2.0
Queue-Service - v3.0.10
Launcher - v6.0.200
Build Cluster Worker - v3.1.1
Contributors
Thanks to the following contributors for making this feature possible:
Akinori
Alan
foxtrot0304
Keisuke
Ming
Pritam
Sagar
Tiffany
Vonny
Yuki
Yuta
Questions and Suggestions
We’d love to hear from you. If you have any questions, please feel free to reach out here. You can also visit us on Github and Slack.
The Stages feature in Screwdriver lets users group jobs with similar objectives, simplifying workflow organization. Jobs within a stage can run sequentially or in parallel based on specific triggers and can include setup and teardown tasks. This categorizes and organizes jobs, managing complex workflows efficiently. For more details, visit the Stages Documentation.
Pipeline Templates
Pipeline Templates in Screwdriver let users define reusable pipeline configurations with predefined jobs, steps, and container images. Users can find, use, and version existing templates, or create new ones in sd-template.yaml. Templates can be tested, validated, tagged, and published through the Screwdriver pipeline, streamlining setup and ensuring consistency. For more details, visit the Pipeline Templates Documentation.
Add the builds endpoint for efficient build retrieval per pipeline.
This update introduces the GET /pipelines/:id/builds endpoint, allowing users to fetch the latest pipeline builds with parameters like groupEventId and latest. This reduces API calls and improves build retrieval efficiency.
Add filter to Jobs Endpoint for PR Job Retrieval in a Pipeline
This PR introduces a filter for the GET /pipelines/:id/jobs endpoint to fetch only pull request (PR) jobs by adding a type=pr query parameter. This change aims to reduce the number of API calls needed to gather information about PR jobs, enhancing efficiency.
Add SHA filter for Pipeline Events
This change introduces a new feature that adds a SHA filter for pipeline events in Screwdriver. This enhancement allows users to filter pipeline events based on specific commit SHAs, providing more granular control and visibility into pipeline executions tied to particular commits.
Fixes
Return 403/404 Errors for Mismatched SCM Host
This PR updates the API to return appropriate error codes when attempting to stop events with a mismatched SCM host. Instead of a generic 500 error, it will now return a 403 error for public pipelines or a 404 error for private pipelines, aligning with the intended security and access control measures.
Subscribed SCM Notifications to Respect Event Types and Allow PR Jobs of Upstream Pipelines
This pull request fixes issues with SCM notifications, ensuring they respect event types and allow PR jobs from upstream pipelines. The correction involves preserving the original SHA of the upstream pipeline and matching event types between the upstream repository and downstream subscriptions, improving functionality and reliability.
Improve Visibility of Build Parameters in Modal
This PR addresses an issue with build parameters being hidden when displayed in a dropdown. The update ensures the screen scrolls automatically to make the entire dropdown visible.
Indicate Pipeline Execution by Pipeline Token
The implementation aims to make it clear when a pipeline is executed using a pipeline token. Currently, it appears as if a pipeline admin started it, causing confusion. The improvement will explicitly indicate execution by a pipeline token, enhancing transparency and user understanding.
Refactor Build Triggers for Improved Readability and Maintainability
This issue highlights the need to refactor trigger processing in the Screwdriver build system due to code complexity, which hampers debugging and maintenance. The proposed changes involve creating a dedicated directory for triggers, splitting processes into separate files, and simplifying complex functions. The goal is to improve readability, maintainability, testing ease, and the reliability of build triggers.
Correct Template Name Formatting on Usage Page
This PR corrects the template name format on the usage page of the Screwdriver CD UI. It changes the display from <name>/<namespace> to <namespace>/<name> to ensure the correct naming convention is used.
Improve Notification Handling for Build Status Transitions
This PR fixes an issue in the Screwdriver CD UI to prevent unnecessary notifications when transitioning between builds via the COMMIT drop-down. Previously, notifications were triggered if the statuses of the builds being navigated to differed. The update ensures that these extra notifications are avoided, improving user experience.
These pull requests (PRs) include several small fixes and improvements to the Screwdriver UI.
PR #1030: Fixed height on PR list element
PR #1026: Fixed background-color for profile icon when focused
PR #1025: Reverted logic for combining graph data
PR #1024: Fixed background color on hover for profile icon
PR #1023: Updated pipeline events page style
PR #1022: Updated nav banner style
PR #1020: Set application navbar to a constant height
PR #1064: Fixed the job list view that was missing in Safari browser
Compatibility List
In order to have these improvements, you will need these minimum versions as listed below.
API - v7.0.140
UI - v1.0.947
Store - v5.0.10
Queue-Service - v3.0.10
Launcher - v6.0.200
Build Cluster Worker - v3.1.1
Author
Vonny Jap, Sr Mgr, Software Dev Engineering, Yahoo
Update Node Start and Restart Logic in Screwdriver UI
This implementation aims to enhance the tooltip functionality in the Screwdriver UI by updating the logic to depend solely on workflow graph information for determining job start and restart capabilities. This approach addresses issues caused by missing data from the API, ensuring tooltips are correctly generated based on the current view and associated workflow graph details.
Fix Padding on Pipeline Routes to Remove Whitespace
This pull request resolves an issue with the left and right padding on pipeline routes in the Screwdriver UI. The excessive padding was causing unwanted whitespace, which affected the overall layout and user experience.
Standardize Navbar Height for Consistent Layout
This update addresses the issue of the navbar's height changing based on window size, which, while minor, can cause inconsistencies in the layout. By setting the navbar to a constant height, we eliminate these variations, simplifying height calculations for other components and ensuring a more consistent user experience.
Features
Implement Browser Notifications
The implementation includes enhancements to the notification system through the browsers to ensure timely and relevant updates for users. This feature aims to provide a more intuitive and informative user experience by optimizing the delivery and content of notifications based on user actions and system events.
Include job template information on the build detail page
In Screwdriver, jobs can implement existing templates. This pull request modifies the build detail page to display information about the template used by each build, along with a link to view details about the template itself.
Author
Vonny Jap, Sr Mgr, Software Dev Engineering, Yahoo
Retroactive: February 2024 Release Rollout and Bug Bash
We are currently catching up with retro updates, so we kindly ask for your patience as we return to our normal rhythm.
Fixes
Fix PR Build Configuration to Ignore Periodic Triggers
This pull request addresses an issue where pull request (PR) jobs were inadvertently triggering periodic builds. The fix involves adding a check to ensure that PR jobs do not start periodic builds. This change prevents unnecessary builds and ensures that the CI/CD pipeline operates as intended without running redundant periodic jobs for PRs.
Fix PR Graph Display for Non-ChainPR Configurations
This pull request addresses an issue where the pipeline graph for pull requests is not displayed correctly when the chainPR setting is set to false. The update ensures that the pipeline graph is rendered accurately for these configurations, and also makes the graph nodes fully clickable with accessible dropdown menu options. This fix improves the usability and accuracy of the PR view in the Screwdriver UI.
Fix PR Graph Rendering Issue When Switching to PR Tab
This pull request fixes a bug that affects the rendering of the PR graph when switching from the Events view to the Pull Request view in the Screwdriver UI. The issue occurs particularly with pipelines containing extensive build data and the chainPR setting enabled. The fix ensures that the showDownstreamTriggers field is correctly set to false when the Pull Request view is rendered, allowing for proper display of the pipeline graphs.
Features
Add Resource Usage Information in the Build Launcher
This pull request adds resource usage information to the metrics pushed to Prometheus. The objective is to capture and report the amount of resources (such as CPU and memory) used by builds, providing better insights into resource consumption.
Enhance Validator Page with Toggle for Split and Spread Views
This pull request enhances the validator page's user interface by adding a button to toggle between split view and spread view. This feature addresses the issue of narrow display areas for the validator input form, workflow graph, and job details, particularly for users with vertical monitors. The spread view option allows for a more expansive and user-friendly layout, improving usability and accessibility.
Author
Vonny Jap, Sr Mgr, Software Dev Engineering, Yahoo
Retroactive: January 2024 Release Rollout and Bug Bash
Features
Automatic Pipeline Data Refresh Post-Synchronization in Screwdriver UI
It introduces a feature that automatically reloads pipeline data after the "sync pipeline" button is clicked, ensuring that the UI reflects the latest pipeline information without requiring a manual page refresh.
Add no-image-pull Option to Skip Docker Image Pull in sd-local
This introduces a new build option called no-image-pull. This option allows users to skip the Docker image pull step when running builds, which can significantly reduce build times when the same image is used repeatedly.
Fixes
Automatic Loading and Start/Stop Toggle for PR Builds in Screwdriver UI
This update fixes a bug where pull request (PR) builds were not automatically loading when the "start" button was clicked. It also ensures that PR builds load automatically, and the button changes to "stop" while the builds are running.
Fix breakage on Options Page with Long Job Names in Screwdriver UI
This fixes the UI issue where long job names caused screen breakage on the Options page. The update truncates long job names with an ellipsis and adds a tooltip to display the full name on hover. Additionally, it includes functionality to expand and collapse long messages for better readability without breaking the layout.
Author
Vonny Jap, Sr Mgr, Software Dev Engineering, Yahoo
Screwdriver team is pleased to announce our newest release which brings in new features and bug fixes across various components.
New Features
UI
Enable cluster admin to force restrictPR setting
Artifact viewer to support displaying Single Page Applications
Display warnings generated during a build.
Enhanced Template page to display usage stats for individual versions.
Bug Fixes
UI
Continue fixing issues related to ember.js upgrade
API
Fix for duplicate events created during remote join
Fix for a security issue where an attacker can assume admin privileges of a previous SD admin who has given up their Github id. This change is applicable to all supported SCMs (Github, Gitlab and Bitbucket)
Fix authorization checks for Bitbucket SCM integration
Queue Service
Fix for more than one periodic build running at the same time
Internals
Command upload to use streams for improved performance
Compatibility List
In order to have these improvements, you will need these minimum versions:
API - v6.0.34
UI - v1.0.820
Store - v5.0.7
Queue-Service - v3.0.6
Launcher - v6.0.182
Build Cluster Worker - v3.0.3
Contributors
Thanks to the following contributors for making this feature possible:
Alan
Haruka
Ibuki
Keisuke
Kevin
Pritam
Sagar
Yuki
Yuta
Questions and Suggestions
We’d love to hear from you. If you have any questions, please feel free to reach out here. You can also visit us on Github and Slack.
Happy Holidays! Screwdriver team is pleased to announce our newest release which brings in new features and bug fixes across various components.
New Features
UI
Enable deleting disconnected Child Pipelines from UI. This will give users more awareness and control over SCM URLs that are removed from child pipelines list.
API
Cluster admins can configure different bookends for individual build clusters.
Add more audit logs for Cluster admins to track API usage.
Bug Fixes
UI
Collections sorting enhancements.
Create Pipeline flow now displays all Templates properly.
API
Pipeline badges have been refactored to reduce resource usage..
Prevent artifact upload errors due to incorrect retry logic.
Queue Service
Prevent archived jobs from running periodic jobs if cleanup fails at any point.
Internals
Update golang version to 1.19 across all golang projects.
Node.js has been upgraded to v18 for Store, Queue Service & Build Cluster Worker.
Feature flag added to Queue Service to control Redis Table usage to track periodic builds.
Compatibility List
In order to have these improvements, you will need these minimum versions:
API - v5.0.12
UI - v1.0.759
Store - v5.0.2
Queue-Service - v3.0.0
Launcher - v6.0.178
Build Cluster Worker - v3.0.2
Contributors
Thanks to the following contributors for making this feature possible:
Alan
Anusha
Kevin
Haruka
Ibuki
Masataka
Pritam
Sagar
Tiffany
Yoshiyuki
Yuki
Yuta
Questions and Suggestions
We’d love to hear from you. If you have any questions, please feel free to reach out here. You can also visit us on Github and Slack.
Author
Jithin Emmanuel, Director Of Engineering, Yahoo
Happy Halloween! Screwdriver team is pleased to announce our newest release which brings in new features and bug fixes across various components.
New Features
Add sorting on branch and status for Collections
Able to select timestamp format in user preferences
Click on User profile in upper right corner, select User Settings
Select dropdown for Timestamp Format, pick preferred format
Click Save
Soft delete for child pipelines - still need to ask a Screwdriver admin to remove completely
Notify Screwdriver pipeline developers if pipeline is missing admin
Add audit log of operations performed on the Pipeline Options page - Screwdriver admins should see more information in API logs
API to reset user settings
Support Redis cluster connection
Add default event meta in launcher - set event.creator properly
New gitversion binary with multiple branch support - added homebrew formula and added parameter --merged (to consider only versions on the current branch)
Bug Fixes
UI
Show error message when unauthorized users change job state
Job state should be updated properly for delayed API response
Gray out the Restart button for jobs that are disabled
Modify toggle text to work in both directions
Display full pipeline name in Collections
Allow reset of Pipeline alias
Remove default pipeline alias name
Add tooltip for build history in Collections
API
Admins can sync on any pipeline
Refactor unzipArtifactsEnabled configuration
Check permissions before running startAll on child pipelines
ID schema for pipeline get latestBuild
Internals
Models
Refactor syncStages to fail early
Pull Request sync only returns PRs relevant to the pipeline
Add more logs to stage creation
Data-schema
Display JobNameLength in user settings
Remove old unique constraint for stages table
SCM GitHub
Get open pull requests - override the default limit (30) to return up to 100)
Change wget to curl for downloading sd-repo
Builds cannot be started if a pipeline has more than 5 invalid admins
Coverage-sonar
Use correct job name for PR with job scope
Queue-Service
Remove laabr
Launcher
Update Github link for grep
Update build status if SIGTERM is received - build status will be updated to Failure when soft evict. Then buildCluster-queue-worker can send a delete request to clean up the build pod
Compatibility List
In order to have these improvements, you will need these minimum versions:
API - v4.1.297
UI - v1.0.732
Store - v4.2.5
Queue-Service - v2.0.42
Launcher - v6.0.171
Build Cluster Worker - v2.24.3
Contributors
Thanks to the following contributors for making this feature possible:
Alan
Anusha
Kevin
Haruka
Ibuki
Masataka
Pritam
Sagar
Sheridan
Shota
Tiffany
Yoshiyuki
Yuki
Yuta
Questions and Suggestions
We’d love to hear from you. If you have any questions, please feel free to reach out here. You can also visit us on Github and Slack.
Author
Tiffany Kyi, Sr Software Dev Engineer, Yahoo
Screwdriver team is pleased to announce our newest release which brings in new features and bug fixes across various components.
New Features
Collections supports sorting by: last time a job was run in a pipeline or build history based on number of failed events/jobs. To sort by one of these fields, click the up/down caret to the right of the field names.
Collections supports displaying a human-readable alias for a Pipeline (in List view). To set the alias for a pipeline, go to your pipeline Options tab. Under Pipeline Preferences, type the alias in the Rename pipeline field. Hit enter. Go to your Collections dashboard to see the new alias.
Screwdriver Admins can perform Sync on any pipeline from the pipeline options UI
If there is no pipeline admin, periodic build jobs will not run and Screwdriver will notify(if Slack or email notifications are configured)
Pull Request Comments are now supported from individual PR jobs
Support for self-hosted SonarQube for individual Pipelines
Meta CLI
Meta CLI can now be installed as homebrew formula
Allow shebang lua commands to have parameters with dashes in them
Updates
User preference to display job name length has now been moved under User Settings. Now you can configure your preference globally for all pipelines. Click on your username in the top right corner to show the dropdown, select User Settings. (Alternatively, navigate directly to https://YOUR_URL/user-settings/preferences). Under the User Preferences tab, click the arrows or type to adjust preferred Display Name Length.
Before:
After:
Bug Fixes
API
Pull Requests jobs added via a pull request should work
Prevent disabled Pull Request jobs from executing
Prevent API crash for Pipelines with large number of Pull Requests
queue-service
Prevent periodic jobs getting dropped due to API connection instabilities and improve error handling
UI
Even in PR workflow-graph job states show up
Build not found redirects to intended pipeline page
Improve the description of the parameter
More consistent restart method when using listViewDisplay message when manually executing jobs for non-latest events
Emphasize non-latest sha warning when manually executing jobs
Use openresty as base image for M1 use
Show error message when unauthorized users change job state
Gray out restart button for jobs that are disabled
Modify toggle text to work in both directions
Collections and pipeline options improvements
Launcher:
Add SD_STEP_NAME env variable
Internals
sd-cmd:
Create command binary atomically
Add configuration to README.md, local configuration improvements
Fix sd-cmd not to slurp all input
buildcluster-queue-worker:
Upgrade amqplib from 0.8.0 to 0.10.0
Compatibility List
In order to have these improvements, you will need these minimum versions:
API - v4.1.282
UI - v1.0.718
Store - v4.2.5
Queue-Service - v2.0.40
Launcher - v6.0.165
Build Cluster Worker - v2.24.3
Contributors
Thanks to the following contributors for making this feature possible:
Alan
Anusha
Haruka
Ibuki
Jacob
Jithin
Kazuyuki
Keisuke
Kenta
Kevin
Naoaki
Pritam
Sagar
Sheridan
Tatsuya
Tiffany
Yoshiyuki
Yuichi
Yuki
Yuta
Questions and Suggestions
We’d love to hear from you. If you have any questions, please feel free to reach out here. You can also visit us on Github and Slack.
In any file ending in .lua with the executable bit set (chmod a+x), putting a "shebang" line like the following lets you run it, and even pass arguments to the script that won't be swallowed by meta
hello-world.lua
#!/usr/bin/env meta print("hello world")
Screwdriver's meta tool is provided to every job, regardless of which image you choose.
This means that you can write Screwdriver commands or helper scripts as Lua programs.
It was inspired by (but unrelated to) etcd's bolt, as meta is a key-value store of sorts, and its boltcli which also provides a lua runner that interfaces with bolt.
Example script or sd-cmd
run.lua
#!/usr/bin/env meta meta.set("a-plain-string-key", "somevalue") meta.set("a-key-for-json-value", { name = "thename", num = 123, array = { "foo", "bar", "baz" } })
What is included?
A Lua 5.1 interpreter written in go (gopher-lua)
meta CLI commands are exposed as methods on the meta object
Libraries (aka "modules") included in gopher-lua-libs - while there are many to choose from here, some highlights include:
argparse - when writing scripts, this is a nice CLI parser inspired from the python one.
Encoding modules: json, yaml, and base64 allow you to decode or encode values as needed.
String helper modules: strings, and shellescape
http client - helpful if you want to use the Screwdriver REST API possibly using os.getenv with the environment vars provided by screwdriver - SD_API_URL, SD_TOKEN, SD_BUILD_ID can be very useful.
plugin - is an advanced technique for parallelism by firing up several "workers" or "threads" as "goroutines" under the hood and communicating via go channels. More than likely overkill for normal use-cases, but it may come in handy, such as fetching all artifacts from another job by reading its manifest.txt and fetching in parallel.
Why is this interesting/useful?
meta is atomic
When invoked, meta obtains an advisory lock via flock.
However, if you wanted to update a value from the shell, you might perform two commands and lose the atomicity:
# Note, to treat the value as an integer rather than string, use -j to indicate json declare -i foo_count="$(meta get -j foo_count)" meta set -j foo_count "$((++foo_count))"
While uncommon, if you write builds that do several things in parallel (perhaps a Makefile run with make -j $(nproc)), making such an update in parallel could hit race conditions between the get and set.
Instead, consider this script (or sd-cmd)
increment-key.lua
#!/usr/bin/env meta local argparse = require 'argparse' local parser = argparse(arg[0], 'increment the value of a key') parser:argument('key', 'The key to increment') local args = parser:parse() local value = tonumber(meta.get(args.key)) or 0 value = value + 1 meta.set(args.key, value) print(value)
The meta tool is made available to all builds, regardless of the image your build chooses - including minimal jobs intended for fanning in several jobs to a single one for further pipeline job-dependency graphs (i.e. screwdrivercd/noop-container)
Screwdrivers commands can help share common tasks between jobs within an organization. When commands are written in bash, then any callouts it makes such as jq must either exist on the images or be installed by the sd-cmd. While writing in meta's lua is not completely immune to needing "other things", at least it has proper http and json support for making and interpreting REST calls.
running "inside" meta can workaround system limits
Occasionally, if the data you put into meta gets very large, you may encounter Limits on size of arguments and environment, which comes from UNIX systems when invoking executables.
Imagine, for instance, wanting to put a file value into meta (NOTE: this is not a recommendation to put large things in meta, but, on the occasions where you need to, it can be supported). Say I have a file foobar.txt and want to put it into some-key. This code:
foobar="$(< foobar.txt)" meta set some-key "$foobar"
May fail to invoke meta at all if the args get too big.
If, instead, the contents are passed over redirection rather than an argument, this limit can be avoided:
load-file.lua
#!/usr/bin/env meta local argparse = require 'argparse' local parser = argparse(arg[0], 'load json from a file') parser:argument('key', 'The key to put the json in') parser:argument('filename', 'The filename') local args = parser:parse() local f, err = io.open(args.filename, 'r') assert(not err, err) local value = f:read("*a") -- Meta set the key to the contents of the file meta.set(args.key, value)
May be invoked with either the filename or, if the data is in memory with the named stdin device
# Direct from the file ./load-file.lua some-key foobar.txt # If in memory using "Here String" (https://www.gnu.org/software/bash/manual/bash.html#Here-Strings) foobar="$(< foobar.txt)" ./load-file.lua some-key /dev/stdin <<<"$foobar"
Additional examples
Using http module to obtain the parent id
get-parent-build-id.lua
#!/usr/bin/env meta local http = require 'http' local json = require 'json' SD_BUILD_ID = os.getenv('SD_BUILD_ID') or error('SD_BUILD_ID environment variable is required') SD_TOKEN = os.getenv('SD_TOKEN') or error('SD_TOKEN environment variable is required') SD_API_URL = os.getenv('SD_API_URL') or error('SD_API_URL environment variable is required') local client = http.client({ headers = { Authorization = "Bearer " .. SD_TOKEN } }) local url = string.format("%sbuilds/%d", SD_API_URL, SD_BUILD_ID) print(string.format("fetching buildInfo from %s", url)) local response, err = client:do_request(http.request("GET", url)) assert(not err, err) assert(response.code == 200, "error code not ok " .. response.code) local buildInfo = json.decode(response.body) print(tonumber(buildInfo.parentBuildId) or 0)
Invocation examples:
# From a job that is triggered from another job declare -i parent_build_id="$(./get-parent-build-id.lua)" echo "$parent_build_id" 48242862 # From a job that is not triggered by another job declare -i parent_build_id="$(./get-parent-build-id.lua)" echo "$parent_build_id" 0
Larger example to pull down manifests from triggering job in parallel
This advanced script creates 3 argparse "commands" (manifest, copy, and parent-id) to help copying manifest files from parent job (the job that triggers this one).
it demonstrates advanced argparse features, http client, and the plugin module to create a "boss + workers" pattern for parallel fetches:
Multiple workers fetch individual files requested by a work channel
The "boss" (main thread) filters relevent files from the manifest which it sends down the work channel
The "boss" closes the work channel, then waits for all workers to complete tasks (note that a channel will still deliver any elements before a receive() call reports not ok
This improves throughput considerably when fetching many files - from a worst case of the sum of all download times with one at a time, to a best case of just the maximum download time when all are done in parallel and network bandwidth is sufficient.
manifest.lua
#!/usr/bin/env meta -- Imports argparse = require 'argparse' plugin = require 'plugin' http = require 'http' json = require 'json' log = require 'log' strings = require 'strings' filepath = require 'filepath' goos = require 'goos' -- Parse the request parser = argparse(arg[0], 'Artifact operations such as fetching manifest or artifacts from another build') parser:option('-l --loglevel', 'Set the loglevel', 'info') parser:option('-b --build-id', 'Build ID') manifestCommand = parser:command('manifest', 'fetch the manifest') manifestCommand:option('-k --key', 'The key to set information in') copyCommand = parser:command('copy', 'Copy from and to') copyCommand:option('-p --parallelism', 'Parallelism when copying multiple artifacts', 4) copyCommand:flag('-d --dir') copyCommand:argument('source', 'Source file') copyCommand:argument('dest', 'Destination file') parentIdCommand = parser:command("parent-id", "Print the parent-id of this build") args = parser:parse() -- Setup logs is shared with workers when parallelizing fetches function setupLogs(args) -- Setup logs log.debug = log.new('STDERR') log.debug:set_prefix("[DEBUG] ") log.debug:set_flags { date = true } log.info = log.new('STDERR') log.info:set_prefix("[INFO] ") log.info:set_flags { date = true } -- TODO(scr): improve log library to deal with levels if args.loglevel == 'info' then log.debug:set_output('/dev/null') elseif args.loglevel == 'warning' or args.loglevel == 'warning' then log.debug:set_output('/dev/null') log.info:set_output('/dev/null') end end setupLogs(args) -- Globals from env function setupGlobals() SD_API_URL = os.getenv('SD_API_URL') assert(SD_API_URL, 'missing SD_API_URL') SD_TOKEN = os.getenv('SD_TOKEN') assert(SD_TOKEN, 'missing SD_TOKEN') client = http.client({ headers = { Authorization = "Bearer " .. SD_TOKEN } }) end setupGlobals() -- Functions -- getBuildInfo gets the build info json object from the buildId function getBuildInfo(buildId) if not buildInfo then local url = string.format("%sbuilds/%d", SD_API_URL, buildId) log.debug:printf("fetching buildInfo from %s", url) local response, err = client:do_request(http.request("GET", url)) assert(not err, err) assert(response.code == 200, "error code not ok " .. response.code) buildInfo = json.decode(response.body) end return buildInfo end -- getParentBuildId gets the parent build ID from this build’s info function getParentBuildId(buildId) local parentBuildId = getBuildInfo(buildId).parentBuildId assert(parentBuildId, string.format("could not get parendId for %d", buildId)) return parentBuildId end -- getArtifact gets and returns the requested artifact function getArtifact(buildId, artifact) local url = string.format("%sbuilds/%d/artifacts/%s", SD_API_URL, buildId, artifact) log.debug:printf("fetching artifact from %s", url) local response, err = client:do_request(http.request("GET", url)) assert(not err, err) assert(response.code == 200, string.format("error code not ok %d for url %s", response.code, url)) return response.body end -- getManifestLines returns an iterator for the lines of the manifest and strips off leading ./ function getManifestLines(buildId) return coroutine.wrap(function() local manifest = getArtifact(buildId, 'manifest.txt') local manifest_lines = strings.split(manifest, '\n') for _, line in ipairs(manifest_lines) do line = strings.trim_prefix(line, './') if line ~= '' then coroutine.yield(line) end end end) end -- fetchArtifact fetches the artifact "source" and writes to a local file "dest" function fetchArtifact(buildId, source, dest) log.info:printf("Copying %s to %s", source, dest) local sourceContent = getArtifact(buildId, source) local dest_file = io.open(dest, 'w') dest_file:write(sourceContent) dest_file:close() end -- fetchArtifactDirectory fetches all the artifacts matching "source" from the manifest and writes to a folder "dest" function fetchArtifactDirectory(buildId, source, dest) -- Fire up workers to run fetches in parallel local work_body = [[ http = require 'http' json = require 'json' log = require 'log' strings = require 'strings' filepath = require 'filepath' goos = require 'goos' local args, workCh setupLogs, setupGlobals, fetchArtifact, getArtifact, args, workCh = unpack(arg) setupLogs(args) setupGlobals() log.debug:printf("Starting work %p", _G) local ok, work = workCh:receive() while ok do log.debug:print(table.concat(work, ' ')) fetchArtifact(unpack(work)) ok, work = workCh:receive() end log.debug:printf("No more work %p", _G) ]] local workCh = channel.make(tonumber(args.parallelism)) local workers = {} for i = 1, tonumber(args.parallelism) do local worker_plugin = plugin.do_string(work_body, setupLogs, setupGlobals, fetchArtifact, getArtifact, args, workCh) local err = worker_plugin:run() assert(not err, err) table.insert(workers, worker_plugin) end -- Send workers work to do log.info:printf("Copying directory %s to %s", source, dest) local source_prefix = strings.trim_suffix(source, filepath.separator()) .. filepath.separator() for line in getManifestLines(buildId) do log.debug:print(line, source_prefix) if source == '.' or source == '' or strings.has_prefix(line, source_prefix) then local dest_dir = filepath.join(dest, filepath.dir(line)) goos.mkdir_all(dest_dir) workCh:send { buildId, line, filepath.join(dest, line) } end end -- Close the work channel to signal workers to exit log.debug:print('Closing workCh') err = workCh:close() assert(not err, err) -- Wait for workers to exit log.debug:print('Waiting for workers to finish') for _, worker in ipairs(workers) do local err = worker:wait() assert(not err, err) end log.info:printf("Done copying directory %s to %s", source, dest) end -- Normalize/help the buildId by getting the parent build id as a convenience if not args.build_id then SD_BUILD_ID = os.getenv('SD_BUILD_ID') assert(SD_BUILD_ID, 'missing SD_BUILD_ID') args.build_id = getParentBuildId(SD_BUILD_ID) end -- Handle the command if args.manifest then local value = {} for line in getManifestLines(args.build_id) do table.insert(value, line) if not args.key then print(line) end end if args.key then meta.set(args.key, value) end elseif args.copy then if args.dir then fetchArtifactDirectory(args.build_id, args.source, args.dest) else fetchArtifact(args.build_id, args.source, args.dest) end elseif args['parent-id'] then print(getParentBuildId(args.build_id)) end
Testing
In order to test this, bats testing system was used to invoke manifest.lua with various arguments and the return code, output, and side-effects checked.
For unit tests, an http server was fired up to serve static files in a testdata directory, and manifest.lua was actually invoked within this test.lua file so that the http server and the manifest.lua were run in two separate threads (via the plugin module) but the same process (to avoid being blocked by meta's locking mechanism, if run in two processes)
test.lua
#!/usr/bin/env meta -- Because Meta locks, run the webserver as a plugin in the same process, then invoke the actual file under test. local plugin = require 'plugin' local filepath = require 'filepath' local argparse = require 'argparse' local http = require 'http' local parser = argparse(arg[0], 'Test runner that serves http test server') parser:option('-d --dir', 'Dir to serve', filepath.join(filepath.dir(arg[0]), "testdata")) parser:option('-a --addr', 'Address to serve on', "localhost:2113") parser:argument('rest', "Rest of the args") :args '*' local args = parser:parse() -- Run an http server on the requested (or default) addr and dir local http_plugin = plugin.do_string([[ local http = require 'http' local args = unpack(arg) http.serve_static(args.dir, args.addr) ]], args) http_plugin:run() -- Wait for http server to be running and serve status.html local wait_plugin = plugin.do_string([[ local http = require 'http' local args = unpack(arg) local client = http.client() local url = string.format("http://%s/status.html", args.addr) repeat local response, err = client:do_request(http.request("GET", url)) until not err and response.code == 200 ]], args) wait_plugin:run() -- Wait for it to finish up to 2 seconds local err = wait_plugin:wait(2) assert(not err, err) -- With the http server running, run the actual file under test -- Run with a plugin so that none of the plugins used by _this file_ are loaded before invoking dofile local run_plugin = plugin.do_string([[ arg[0] = table.remove(arg, 1) dofile(arg[0]) ]], unpack(args.rest)) run_plugin:run() -- Wait for the run to complete and report errors, if any local err = run_plugin:wait() assert(not err, err) -- Stop the http server for good measure http_plugin:stop()
