在线时间:8:00-16:00
迪恩网络APP
随时随地掌握行业动态
扫描二维码
关注迪恩网络微信公众号
开源软件名称:iamduo/workq开源软件地址:https://github.com/iamduo/workq开源编程语言:Go 99.7%开源软件介绍:WorkqWorkq is a job scheduling server strictly focused on simplifying job processing and streamlining coordination. It can run jobs in blocking foreground or non-blocking background mode. Workq runs as a standalone TCP server and implements a simple, text based protocol. Clients interact with Workq over a TCP socket in a request/response model with text commands. Please refer to the full protocol doc for details. Supported Features
What is Workq useful for?Scheduling & Timers Schedule adhoc one-time jobs at a future UTC time. Workq can be used as a timer for any application event scheduling. Compare this to the usual alternative of building scheduling schema into an existing datastore, writing a custom worker to poll the datastore, and finally building in locking & state management to manage dispatching. In Workq, you schedule the job, lease the job, and complete the job. Concurrency Control Workq can enable concurrency especially in languages that do not have built-in concurrency. Background multiple jobs, process them with multiple workers, and retrieve results from within a single process. Streamlining & Persistent Processing Workq will retry jobs from TTR (time-to-run) timeouts and/or from explicit job failures through the max-attempts and max-fails flags. Retry is a job execution specification and does not require any custom worker logic. Distributing Work Distribute work to multiple workers using Workq as the coordinator. In addition, Workq is language agnostic, submit a job from one language and process it in another. Workq can also naturally store job execution results for later retrieval up to the job's TTL. Project StatusWorkq is in alpha status and not yet stable. The protocol may still be subject to small changes. Full unit test coverage, system testing, and fuzz testing are used to ensure maximum coverage and safety throughout development. Workq is currently suitable for development experimentation and evaluation. Stabilization will happen once the test coverage is thorough enough and there no major gaps in the protocol. There will be absolutely no reliance on manual testing where technically possible. Workq will slow bake until ready. Table of Contents
Getting StartedClient/Worker examples are shown using the Go client. Installing
StartingStart an in-memory only workq-server on port 9922.
Workq can be started with persistence by passing the
Please refer to the full Command Log Doc for details. Connectingimport "github.com/iamduo/go-workq"
// ...
client, err := workq.Connect("localhost:9922")
if err != nil {
// ...
} Submitting Jobs - 3 WaysAdd a Job (Background)Add a background job. The result can be retrieved through the "result" command. job := &workq.BgJob{
ID: "6ba7b810-9dad-11d1-80b4-00c04fd430c4",
Name: "ping",
TTL: 60000, // Expire after 60 seconds
TTR: 5000, // 5 second time-to-run limit
Payload: []byte("ping"),
Priority: 10, // @OPTIONAL Numeric priority, default 0.
MaxAttempts: 3, // @OPTIONAL Absolute max num of attempts.
MaxFails: 1, // @OPTIONAL Absolute max number of failures.
}
err := client.Add(job)
if err != nil {
// ...
} Run a Job (Foreground)Run a job and wait for its result. job := &workq.FgJob{
ID: "6ba7b810-9dad-11d1-80b4-00c04fd430c4",
Name: "ping",
TTR: 5000, // 5 second time-to-run limit
Timeout: 60000, // Wait up to 60 seconds for a worker to pick up.
Payload: []byte("ping"),
Priority: 10, // @OPTIONAL Numeric priority, default 0.
}
result, err := client.Run(job)
if err != nil {
// ...
}
fmt.Printf("Success: %t, Result: %s", result.Success, result.Result) Schedule a JobSchedule a job at a UTC time. The result can be retrieved through the "result" command. job := &workq.ScheduledJob{
ID: "6ba7b810-9dad-11d1-80b4-00c04fd430c4",
Name: "ping",
Time: "2016-12-01T00:00:00Z" // Start job at this UTC time.
TTL: 60000, // Expire after 60 seconds
TTR: 5000, // 5 second time-to-run limit
Payload: []byte("ping"),
Priority: 10, // @OPTIONAL Numeric priority, default 0.
MaxAttempts: 3, // @OPTIONAL Absolute max num of attempts.
MaxFails: 1, // @OPTIONAL Absolute max number of failures.
}
err := client.Schedule(job)
if err != nil {
// ...
} Leasing JobsLease the first job within a set of one or more job names with a wait-timeout (milliseconds). // Lease the first available job in "ping1", "ping2", "ping3"
// waiting up to 60 seconds.
job, err := client.Lease([]string{"ping1", "ping2", "ping3"}, 60000)
if err != nil {
// ...
}
fmt.Printf("Leased Job: ID: %s, Name: %s, Payload: %s", job.ID, job.Name, job.Payload) Completing JobsMark a job successfully completed with a result. err := client.Complete("6ba7b810-9dad-11d1-80b4-00c04fd430c4", []byte("pong"))
if err != nil {
// ...
} Failing JobsMark a job failed with a result. err := client.Fail("6ba7b810-9dad-11d1-80b4-00c04fd430c4", []byte("failed"))
if err != nil {
// ...
} Retrieving Job ResultsGet a job result previously executed by Add or Schedule commands. // Get a job result, waiting up to 60 seconds if the job is still executing.
result, err := client.Result("6ba7b810-9dad-11d1-80b4-00c04fd430c4", 60000)
if err != nil {
// ...
}
fmt.Printf("Success: %t, Result: %s", result.Success, result.Result) And MorePlease see the full protocol doc and Go client docs for more details. Command Log PersistencePersistence in Workq is handled by the Command Log, an append-only data log of all job commands that perform state changes. Please refer to the full Command Log Doc for details. ClientsGoJavaDeveloping New ClientsWhile there is no official client development guide yet, developing a Workq client is fairly straightforward simply by reading the protocol doc. In addition, you can review the Go client as a working example. If you have questions, please open a Github issue with the "question" label. Caveats & LimitationsWorkq can't and will not do everything you want. Some things to keep in mind:
GlossaryPriorityPriority is a signed 32-bit integer value from -2147483648 to 2147483647, 0 being the default. Controls the order of job execution within a single job queue. Higher priorities are executed first. TTR - Time-to-runTTR stands for time-to-run and is the maximum time a job can run for after being leased before it is requeued. This value is in milliseconds. TTL - Time-to-liveExpiration time before job is deleted regardless of state. This value is in milliseconds. LeaseA worker receives work by leasing jobs by their name. A lease is bound by the job's TTR. Max-AttemptsMax Attempts is used in Max-FailsMax number of explicit job failures. Explicit job failures are not from timeouts, but from explicit workers reporting results by the Contributing
Contribution SuitabilityThe project maintainer has the last word on whether or not a contribution is suitable for Workq. All contributions will be considered carefully, but from time to time, contributions will be rejected because they do not suit the current goals or needs of the project. If your contribution is rejected, don't despair! As long as you followed these guidelines, you will have a much better chance of getting your next contribution accepted. Bug ReportsBug reports are hugely important! Before you raise one, though, please check through the GitHub issues, both open and closed, to confirm that the bug hasn't been reported before. Duplicate bug reports are a huge drain on the time of other contributors, and should be avoided as much as possible. Feature RequestsWorkq is intended to have a small API to solve a very specific set of use cases. Maintaining a small API allows Workq to be maintainable in the future and understandable. One of the most important skills to have while maintaining an open source project is learning the ability to say "no" to suggested changes, while keeping an open ear and mind. Admittedly Workq is still in its early stages. If you believe there is a feature missing, feel free to raise a feature request, but please do be aware that not all requests are suitable.
Supporting WorkqIf enough users find value in Workq, I'll attempt to set up a strategy to sustain it for the long term. If you find value in Workq for your company, please say hello on our mailing list. CreditsWorkq is inspired from prior works. The following projects have shaped Workq's feature set:
ThanksSpecial thanks to Theofanis M3 Industries, the initial Workq Supporter, which provided me the hardware to build Workq. |
2023-10-27
2022-08-15
2022-08-17
2022-09-23
2022-08-13
请发表评论