#bioc-builds

2018-12-11

Aaron Lun (05:11:10): > @Aaron Lun has joined the channel

Aaron Lun (05:11:11): > set the channel description: A safe space for dealing with BioC build system pain. Suffering shared is suffering halved.

Shian Su (05:12:45): > @Shian Su has joined the channel

Aaron Lun (05:22:55): > ARGH

Aaron Lun (05:22:58): > ARGH

Aaron Lun (05:22:59): > ARGH

Aaron Lun (05:23:11): > :sad-parrot::sad-parrot::sad-parrot::sad-parrot:

Shian Su (05:23:19): > Is this what the channel is going to be?

Aaron Lun (05:23:22): > I have christened this channel with my tears.

Martin Morgan (08:13:39): > @Martin Morgan has joined the channel

Martin Morgan (08:17:30): > It’s great to have community discussion here; if the discussion matures to a question / action for the core team, then please post to the bioc-devel mailing list.

Aaron Lun (08:36:53): > For starters: is it possible to get access to build reports other than the latest? I want to see what changed after the last successful build of BiocNeighbors on Windows.

Martin Morgan (08:39:52): > unfortunately the build reports are over-written nightly, except for special snapshots. It wouldn’t be hard to automatically capture the build reports via BiocPkgTools

Aaron Lun (08:46:11): > Hm. So the BiocNeighbors fail occurred sometime between 6 Nov (the last successful build at 1.1.1) and 1st Dec. The first time I saw it, I remember seeing the R-devel version as being 18 Nov, and I recall someone mentioning that the R-devel version had been updated recently on the build machines. Did the Rtools toolchain also change at the same time?

Aaron Lun (08:54:02): > The old tokay2 builds (shown for BioC <= 3.7) are using-O2 -mtune=generic, while the current tokay2 is using-mtune=native -march=native -O3. The question here is, is the change due to a change in the Rtools default, or did something change with tokay2? And why?

Martin Morgan (08:55:33): > We would have updated R to R-devel, and Rtools to the most recent tool chain; the flags would have (almost certainly) been those out-of-the-box, so if they changed in R-devel, then they changed for us.

Martin Morgan (09:02:57): > These flags should be coming from the R source, probably src/gnuwin32/fixed/Makefile and src/gnuwin32/fixed/etc/Makeconf but it’s hard to figure this out…

Aaron Lun (09:03:29): > The R-devel NEWS certainly isn’t helpful.

Aaron Lun (09:03:50): > Interestingly Rhub also claims to use R-devel on one of its builders but still has-O2 -mtune=generic.

Martin Morgan (09:05:23): > The definitive answer for us (‘defaults’ or ‘weird bioc customization’) is going to come from posting to the bioc-devel mailing list.

Aaron Lun (09:19:51): > I’ll do that tomorrow. In the meantime, I’ll note that a grep of the R-devel source indicates no-marchspecification, no relevant-mtunespecs, and possibly relevant-O3specs.

Valerie Obenchain (09:26:53): > @Valerie Obenchain has joined the channel

Valerie Obenchain (10:24:24): > @Aaron LunI would really appreciate it if you could post build system questions on bioc-devel. The number of people subscribed to bioc-community slack, let alone this new channel, is a small subset of users overall. > > One of the requirements in the new package review process is that new maintainers subscribe to bioc-devel so they can follow relevant devel discussions. IMO finding project information is quite fractured as is with github issues, support site, and bioc-devel. The growing number of channels on bioc-community slack makes this even more challenging for new users trying to stay informed. > > We use Rtools 3.4.0 on both build machines. Compiler flags for the Windows machines are here:https://www.bioconductor.org/checkResults/release/bioc-LATEST/tokay1-NodeInfo.htmlhttps://www.bioconductor.org/checkResults/devel/bioc-LATEST/tokay2-NodeInfo.htmlThe flags for C++14 are not on that listing and probably should be. The current tool chain does not fully support C++14 flags so we are “faking it” by using a work around which I was testing it on tokay2: > > CXX14 = $(BINPREF)g++ $(MULTI_ARCH) > CXX14STD = -std=gnu++14 > > On tokay1 these variables aren’t set: > > CXX14 = > CXX14STD = > > CXX14FLAGSis not set on either builder but maybe it should be whenCXX14andCXX14STDare set, e.g., for consistency, > > CXX14FLAGS = -O2 -Wall $(DEBUGFLAG) -mtune=generic > > I don’t have the exact date I made this change but it was around the week of Nov 19th. Today I’ll setCXX14FLAGSon tokay2 and we can see if they makes a difference for BiocNeighbors.

Valerie Obenchain (10:55:18): > A little more background - > The change was made on tokay2 after conversations with Jeroen Ooms and the knowledge that the CRAN builders do something similar:https://github.com/r-windows/r-testing/issues/1#issuecomment-437631844rtools40 does fully support C++14 so this workaround can be removed when that tool chain is installed.

Aaron Lun (17:38:57): > Thanks@Valerie Obenchain. I’ll post any concrete questions to the mailing list in the future.

Shian Su (17:43:44): > I feel like slack is more appropriate for the shot-in-the-dark debugging currently going on.

Shian Su (17:44:12): > It would be nice the summarise the results of the debugging to the mailing list to help future devs though.

Shian Su (17:48:25): > I want to know if the differences between RcppAnnoy and BiocNeighbors occured at the R or C++ level.

Shian Su (17:49:02): > Are they both calling the same C++ function and can you verify that the matrices are identical upon entry?

Aaron Lun (18:26:24): > Well, I did have some concrete questions in the end, so that was good.

Aaron Lun (18:29:16): > I would say the differences are probably occurring at the C++ level, as there’s not much R code in these functions. And in fact, all other builds are okay, so if it were a R-level difference, you’d expect to see failures across all machines.

Aaron Lun (18:29:35): > That said… if it is a compilation problem, then all bets are off.

Aaron Lun (19:02:02): > Well, now scater fails on windows as well.http://bioconductor.org/checkResults/devel/bioc-LATEST/scater/tokay2-checksrc.html

Aaron Lun (19:02:25): > With differences between 32-bit and 64-bit, which suggests it’s a numerical precision problem in Rtsne.

Shian Su (19:03:10): > Well if there was something numerical relating to how R was compiled then there isn’t much you can do on your end.

Aaron Lun (19:04:22): > Compilation settings that change the output shouldn’t be on inO3.

Aaron Lun (19:05:41): > But the discrepancy could only arise if the Euclidean distance calculations in Rtsne do not match the Euclidean distance calculations in BiocNeighbors.

Aaron Lun (19:06:04): > Which shouldn’t be possible, because the calculation is pretty simple!

Shian Su (19:14:04): > Anything is possible in C++ land

Shian Su (19:16:05): > Something likediff = x1[i] - x2[i]; dist += diff * diffcan produce different results fromdist += pow(x1[i] - x2[i], 2)

Shian Su (19:17:38): > There’s also promotion of temporary doubles to 80 bit precision in x64 architectures. (https://en.wikipedia.org/wiki/Extended_precision#IEEE_754_extended_precision_formats)

Shian Su (19:19:54): > From the compiler dev’s perspective people usually aren’t bothered by 0.001% differences in their numerical results. Those that are will need to enable a bunch of stability options that slow everything down drastically.

Shian Su (19:21:43): > Unfortunately in this case you’re also at the mercy of how R and every other package in the library are built.

Shian Su (19:22:40): > I guess you can try switching your unit tests to be based on % deviation and maybe even tolerate a small number of significantly divergent values.

Aaron Lun (20:50:08): > Nope, the calculations areexactlythe same. As I’ve looked at both the Rtsne and BiocNeighbbors src.

Aaron Lun (20:51:18): > And while compilers may or may not produce the same results between architectures, they should do for the same architecture, which is currently the problem.

Aaron Lun (20:52:04): > The problem is not with the tolerance itself, which I could handle by just using expect_equal. The problem is that the divergence results in changes to the integer identities of the neighbors.

Shian Su (21:16:32): > One workaround is to sayexpect_true(sum(res != expect) < 10)

Aaron Lun (21:17:28): > I’m not sure that solves much.

Shian Su (21:19:01): > It’s accepting that you are using an extremely sensitive algorithm and putting a tolerance on the number of discordant neighbour assignments.

Aaron Lun (21:19:38): > I’d rather just disable the tests altogether for windows.

Shian Su (21:21:46): > Up to you

Shian Su (21:22:14): > Personally I think 5 mis-assignments out of 5000 is still fit for purpose, and the tests could reflect as such.

Aaron Lun (21:25:15): > Well, they’re not misassignments because it’s an approximate method in the first place. The problem is that the nature of the approximation differs between binaries, probably due to compilation differences. I don’t like it and won’t condone it by changing the tests.

Shian Su (21:31:24): > Fair enough, then let’s pray that R-devel has some good ideas.

Aaron Lun (21:54:00): > One can only hope.

2018-12-14

Peter Hickey (21:27:12): > @Peter Hickey has joined the channel

2018-12-15

Aaron Lun (19:00:37): > <!channel>Moving the BiocNeighbors discussion to the mailing list.

Aaron Lun (20:09:08): > Actually, hold that thought.

Aaron Lun (20:09:09): > I’ve done it!

Aaron Lun (20:09:41): > I’ve reproduced the error locally with-O3 -march=native -mtune=native!

Aaron Lun (20:14:10): > So it is an optimization bug. Gotta think of how to debug this.

Aaron Lun (20:35:20): > Narrowed it down to-march=native -mtune=native,-O3by itself is fine.

Aaron Lun (20:43:50): > Oh, funnily enough,-march=nativeis also fine by itself. It’s just a combination of O3 with march=native that fails. Hm.

2018-12-16

Lluís Revilla (08:37:59): > @Lluís Revilla has joined the channel

2018-12-20

Shian Su (21:35:35): > I am so curious to what kind of code wrote a Makevars.win to ~/.R

2018-12-23

Aaron Lun (01:22:41): > Does anyone know if SnowParam on Bioc-devel build dispatches jobs to machines that have different numerical precision in the float calculations? Trying to figure out mysterious test failures fromscranthat have started occuring since the DelayedArray parallel%*%update.

Martin Morgan (02:05:42): > there is only one build machine per architecture so same precision

Aaron Lun (05:54:23): > Hm. Well, I’m stumped again then. I’ll double-check whether this is a problem with the parallelization…

2019-01-02

Stevie Pederson (22:18:31): > @Stevie Pederson has joined the channel

2019-01-03

Aedin Culhane (13:03:37): > @Aedin Culhane has joined the channel

2019-01-08

Valerie Obenchain (09:49:56): > @Valerie Obenchain has left the channel

2019-01-09

Lori Shepherd (06:55:40): > @Lori Shepherd has joined the channel

2019-01-10

Samuela Pollack (09:16:27): > @Samuela Pollack has joined the channel

2019-01-11

Aaron Lun (20:23:22): > new build machinecelaya2!

Aaron Lun (20:25:10): > I thought they were all wine based, but I don’t get this new reference.

2019-01-12

Martin Morgan (08:39:25): > Macs are cities in Mexico. I don’t know why.

Kevin Rue-Albrecht (11:42:27): > @Kevin Rue-Albrecht has joined the channel

Kevin Rue-Albrecht (11:43:34): > Something wrong withscRNAseqpackage ontokay2? > > Error: processing vignette 'basic.Rmd' failed with diagnostics: > there is no package called 'scRNAseq' > > http://bioconductor.org/checkResults/devel/bioc-LATEST/iSEE/tokay2-buildsrc.html

Aaron Lun (20:43:43): > It’ll get fixed.

2019-01-17

Kayla Interdonato (08:27:55): > @Kayla Interdonato has joined the channel

2019-01-24

Aaron Lun (12:57:44): > Yet another BiocNeighbors failure on Windows. I kind-of know where it comes from, as I did some extensive updates to the package recently. The immediate thing that I’d like to know is - what happens when a CRAN package doesn’t yet have Windows binaries? Does it get compiled, and with the same settings as Bioc packages? FYI the one I’m looking at ishttps://cran.r-project.org/web/packages/RcppHNSW/index.html(which now has binaries, but didn’t at the time of the last BioC build).

Martin Morgan (13:16:16): > The Bioc builder acts as a ‘user’ of CRAN packages, installing the Windows binary built however CRAN built it. Both repositories probably aim for a ‘best practice’ of using Rtools ‘out of the box’, though as we saw funky things can happen…

Aaron Lun (13:23:12): > I’ll cross my fingers for the next BioC build. Hopefully the CRAN binaries do the trick.

2019-01-25

Aaron Lun (13:01:31): > Well, that didn’t work. But at least both 32 and 64 are segfaulting.

Aaron Lun (13:14:53): > Ah. buffer overflow ahoy.

Aaron Lun (13:15:17): > Hm. This really shakes my faith invalgrind’s reliability.

Aaron Lun (13:23:21): > No, wait,--use-valgrinddidactually report it. I just had to drill down into the exact test*.Routfile in which it occurred.

Aaron Lun (13:23:38): > That’s more reassuring. Okay, let’s see how this goes.

Aaron Lun (13:24:55): > On an unrelated note:http://bioconductor.org/checkResults/devel/bioc-LATEST/scran/tokay2-checksrc.htmlis giving weird permission errors.

2019-01-29

Aaron Lun (13:30:07): > Hooray! BiocNeighbors is all green.

Aaron Lun (13:30:12): > Butscranis still weird:http://bioconductor.org/checkResults/devel/bioc-LATEST/scran/tokay2-checksrc.html

Aaron Lun (13:30:22): > Some permission errors… any ideas@Martin Morgan?

2019-02-06

Aaron Lun (07:51:41): > > Running examples in ‘scran-Ex.R’ failed > > Warning in file(con, “r”) : > > cannot open file ‘../scran-Ex_i386.Rout’: Permission denied > > Error in file(con, “r”) : cannot open the connection > > Execution halted

Aaron Lun (07:51:54): > Don’t really know what’s happening here.

Martin Morgan (10:09:04) (in thread): > Maybe@Hervé Pagèscan lend some insight

2019-02-16

Aaron Lun (12:36:03) (in thread): > Any thoughts@Hervé Pagès?

2019-02-20

Aaron Lun (06:54:50) (in thread): > Ah, he’s on holiday…

2019-02-26

Kevin Rue-Albrecht (14:50:01): > Are the builds less frequent for the release branch, or is there some kind of lag recently? > - Release (3.8): This page was generated on 2019-02-23 11:44:03 -0500 (Sat, 23 Feb 2019). > - Devel (3.9): This page was generated on 2019-02-26 14:10:42 -0500 (Tue, 26 Feb 2019).

2019-02-27

Lori Shepherd (07:13:34): > there is an issue with the release builds - I am looking into today, if I can’t figure it out, it may not get fixed until fri

Kevin Rue-Albrecht (07:44:21): > Thanks for the update and good luck !

2019-02-28

Kevin Rue-Albrecht (06:06:24): > Friendly follow-up: I assume this issue wasn’t cracked yet?

Lori Shepherd (07:23:13): > We are hoping a build report posts today - still working on it an keeping an eye on it

Kevin Rue-Albrecht (07:25:48): > Thanks! I’ve got no idea how complex the system is, but in any case I appreciate your effort on it:slightly_smiling_face:

2019-03-01

Lori Shepherd (07:27:52): > Looks like they are back up now

Kevin Rue-Albrecht (07:48:47): > awesome, i can see it now!

Kevin Rue-Albrecht (07:48:59): > thanks!

2019-03-06

Aaron Lun (16:39:34): > @Lori ShepherdThis might go away by the time you read it, butscranhas managed to have its reference manual out of sync with the source package in 1.1.20. The reference manual claims thatquickClusterhas an argument namedBSPARAM=, but the source package doesn’t have any such argument. And it ’s not a mismatch between code and docs because the unix build doesn’t show any warnings. I’ve just bumped it to 1.1.21 to fix it, but this might be pointing to something funny with how the webpage interacts with the build system.

Lori Shepherd (16:41:36): > Thanks I’ll check the website builds. Let me know if it doesn’t clear up

Aaron Lun (16:42:20) (in thread): > Now that@Hervé Pagèsis back… any thoughts about this permission problem?

Hervé Pagès (16:42:25): > @Hervé Pagès has joined the channel

Aaron Lun (16:55:33): > And while I’m complaining about things,BiocSingularis still failing because ofMulticoreParamproblems on Windows.@Martin Morganalready fixedBiocParallel, so I don’t know why the build system is acting up here.

2019-03-07

Martin Morgan (15:45:17) (in thread): > Lori and I looked at this more closely. The problem is actually on Windows; previously I think you’d letbplapply()do the start, and it ‘short circuits’ when n == 1 to useSerialParam()and lapply. I think now you explicitly callbpstart(), and I believe this would have failed in release, too. I’ll come up with a workaround…

Hervé Pagès (16:39:24) (in thread): > cannot open file '../scran-Ex_i386.Rout': Permission deniedLooks like a process is still holding on the file so other processes cannot access it.

Hervé Pagès (16:44:24) (in thread): > Continued: note thatR CMD checkfires a separate R process (child process) to run the examples (in that case a 32-bit R process). The output of the child process is captured into a file (scran-Ex_i386.Rout) that the parent process then parses to detect and report errors or significant warnings.

Hervé Pagès (16:55:47) (in thread): > Continued: when the child process returns (i.e. when the 32-bit R session executing the examples is finished) it could be that some process is still holding on the file (scran-Ex_i386.Rout) where the output was being written. This happens typically when the child process itself fired other processes that are also writing to the file. See where I’am going with this? What I suspect is that the examples are using BiocParallel to fire other R sessions and that these other R sessions are still running when the master session (i.e. the direct child ofR CMD check) finishes. As soon as the master session finishes,R CMD checktries to openscran-Ex_i386.Routto parse it but fails because some of its gran-children (which are now orphans) are still holding to it. It could be that those orphans were about to die anyway (in the next micro secs or so) butR CMD checktried to openscran-Ex_i386.Routjust a few micro secs too early.

Hervé Pagès (17:00:39) (in thread): > Continued: it’s called a race condition. Race conditions can be hard to reproduce because ifR CMD checkhad been 2 micro secs slower in openingscran-Ex_i386.Routor if the processes holding toscran-Ex_i386.Routhad died 2 micro secs earlier, everything would have been fine.

Hervé Pagès (17:06:59) (in thread): > Continued: Note that this is a typical Windows problem. We’ve seen it before. Unix-like systems have no problem letting a process open a file that other processes are holding. Also note that this is not a build system problem per-se. The race condition happens duringR CMD check. It doesn’t matter here thatR CMD checkis fired by the build system. I guess one would be able to reproduce this by going on tokay2 and runningR CMD checkinteractively. (I didn’t try this though. Also since this looks like a race condition there is no guarantee that I will be able to reproduce. I might need to try to runR CMD checkseveral times for that).

Hervé Pagès (17:11:58) (in thread): > Continued: Solution is to make sure that you close all the resources that you open in your examples. This means making sure that your workers are done before execution reaches the end of your examples. The BiocParallel experts (@Martin Morgan) might be able to help with this.

Hervé Pagès (17:13:21) (in thread): > Continued: I’m done. Congratulations if you followed me all the way thru that long story:smiley:

2019-03-08

Aaron Lun (01:02:45) (in thread): > right, thanks.

Aaron Lun (01:03:57) (in thread): > oh geez. Maybe I should switch all the examples to use MulticoreParam then, on Windows it just won’t fire.

Aaron Lun (01:04:48) (in thread): > Huh. That’s funny, I’m already using MulticoreParam for all my examples.

Aaron Lun (01:06:13) (in thread): > Must be an implicit bpparam() usage somewhere… probably from DelayedArray%*%.

Aaron Lun (01:09:58) (in thread): > In any case, I’ll wait until BiocSingular builds on Windows; I’m surprised the build system even got to that point without BiocSingular binaries being available.

Hervé Pagès (10:51:18) (in thread): > The Build System installs Bioconductor packages from source (directly fromgit.bioconductor.org) on all platforms. As you can see on the build report, installation of BiocSingular was successful on tokay2.

Aaron Lun (22:43:48) (in thread): > huh. okay.

2019-03-27

Aaron Lun (00:50:30) (in thread): > @Martin MorganThis problem is back (not sure it ever left), though with a slightly different error message from before:http://bioconductor.org/checkResults/devel/bioc-LATEST/BiocSingular/tokay2-buildsrc.html

Martin Morgan (07:02:11) (in thread): > When I thought about this more it seemed basically like a bad idea to make this work – MulticoreParam isn’t supported on Windows, so an error rather than trying to recover. What about, in the vignette > > if (identical(.Platform$OS.type, "unix")) > register(MulticoreParam(2)) > else > register(SerialParam()) > > with brief text explanation, or actually just skipping the specification ofBPPARAM=– the stack of registered back-ends is already appropriate for platform.

2019-03-30

Aaron Lun (02:00:47) (in thread): > Hm. That doesn’t sound good in any context. Analysis scripts using MulticoreParam would simply fail on Windows, for starters. In my package tests, I also specifyBPPARAM=explicitly to check for correct parallelization. Your first proposal makes my code a whole lot more complicated, and your second proposal would mean that I wouldn’t be able to do those tests for any OS.

Aaron Lun (02:06:06) (in thread): > I mean, it seems like we have a choice between the analyst/developer writing all of the boilerplate to choose between MulticoreParam/SerialParam; or just having it done under the hood byBiocParallel.

Aaron Lun (02:50:38) (in thread): > Perhaps I don’t grasp how BP is architectured, but couldn’t you just throw a line into the MulticoreParam constructor that saysif (windows) { nworkers <- 1 }? Surely all of the MCP set-up would just be a no-op if there’s only one core.

Aaron Lun (02:56:58) (in thread): > Well… at least the windows mcparallel errors are now the same between BiocSingular and scran.

2019-04-12

Aaron Lun (23:05:10): > It’s… it’s beautiful. - File (PNG): Pasted image at 2019-04-12, 8:05 PM

2019-04-13

Aaron Lun (02:04:15): > Don’t know if anyone else is seeing this: > > > BiocManager::install("BiocVersion") > Bioconductor version 3.9 (BiocManager 1.30.4), R Under development (unstable) > (2019-04-11 r76379) > Installing package(s) 'BiocVersion' > Warning message: > package 'BiocVersion' is not available (for R Under development) >

Aaron Lun (02:04:29): > > > sessionInfo() > R Under development (unstable) (2019-04-11 r76379) > Platform: x86_64-pc-linux-gnu (64-bit) > Running under: Ubuntu 18.04.2 LTS > > Matrix products: default > BLAS: /home/luna/Software/R/trunk/lib/libRblas.so > LAPACK: /home/luna/Software/R/trunk/lib/libRlapack.so > > locale: > [1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C > [3] LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8 > [5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8 > [7] LC_PAPER=en_US.UTF-8 LC_NAME=C > [9] LC_ADDRESS=C LC_TELEPHONE=C > [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C > > attached base packages: > [1] stats graphics grDevices utils datasets methods base > > loaded via a namespace (and not attached): > [1] BiocManager_1.30.4 compiler_3.7.0 tools_3.7.0 >

Martin Morgan (06:46:58): > You’re using R-devel, but the next release and devel will be using R-3.6 (currently in beta)

Aaron Lun (18:06:36): > Okay, so the switch has already happened.

2019-04-16

Aaron Lun (01:40:44): > I’m getting some impressively nonsensical build failures on Windows (again) for BiocSingular:http://bioconductor.org/checkResults/devel/bioc-LATEST/BiocSingular/tokay2-checksrc.html

Aaron Lun (01:41:56): > The affected tests only fail on Windows, yet the vast majority of them involve pure R code. So it’s not the usual suspects of machine-specific C/C++ behaviour or compilation settings.

Aaron Lun (01:42:30): > I notice that the R-devel revision number on Windows is slightly behind that of the other two build machines - I don’t know if that’s related.

Aaron Lun (01:45:52): > To highlight how weird the test failure is, consider the 32-bit failure on line 46 oftest-lowrank.R. There’s not a lot of magic here - theLowRankMatrixis pretty simple - and if you follow the logic all the way through to the end, it implies thattcrossprod(x, y)does not yield the same numerical result asx %*% t(y). If true, this would contradict the docs, which state that the two approaches are equivalent.

Aaron Lun (01:49:32): > The 64-bit failures seem to imply that theDelayedArrayclass structure is different on Windows compared to the other systems… also weird.

Shian Su (01:54:19): > > * checking package dependencies ...Warning: unable to access index for repository[https://CRAN.R-project.org/src/contrib](https://CRAN.R-project.org/src/contrib): > cannot open URL '[https://CRAN.R-project.org/src/contrib/PACKAGES](https://CRAN.R-project.org/src/contrib/PACKAGES)' > OK > > Didn’t know tokay2 was so sassy.

2019-04-17

Zhi Yang (18:07:46): > @Zhi Yang has joined the channel

2019-04-19

Aaron Lun (18:39:47): > Well… pretty much as I thought!tcrossprod(x, y)yields a different result fromx %*% t(y)on 32-bit tokay whenxis a Matrix andyis an ordinary matrix.

Aaron Lun (18:41:07): > Great. Just great.

Aaron Lun (18:46:11): > I think this says it all:http://bioconductor.org/checkResults/devel/bioc-LATEST/BiocSingular/tokay2-checksrc.html

Martin Morgan (19:15:59): > So that looks like numerical precision, presumably withtcrossprod()being in a better position to perform precise calculations thanrot %*% t(comp)?

Aaron Lun (19:19:34): > Yes, I think so. Reproduced on win-builder, so it’s not just tokay2’s problem.

Aaron Lun (19:20:30): > I would consider this a bug, astcrossprodis meant to give identical results to (but faster than)%*%.

Aaron Lun (19:25:02): > What I find odd is that any differences in summation order should also manifest as inequalities in 64-bit… yet that was fine. So it must be extremely machine specific, probably some madness to do with extended precision calculations on x86.

Aaron Lun (19:43:40): > Well, I put in a bug report to the other Martin M.

Aaron Lun (19:44:16): > Probably nothing to be done except to just add a note about it in the documentation somewhere.

Aaron Lun (19:44:39): > I mean… who uses 32-bit machines nowadays anyway?

2019-05-18

Aaron Lun (01:15:29): > Looks like the citation information disappeared from the devel landing pages…

2019-05-21

Almut (06:35:22): > @Almut has joined the channel

Hervé Pagès (16:51:04) (in thread): > Found the problem. Looks like other things were missing too from some of these pages. They should be fixed soon (in about 1h or so when the updated pages get pushed to the website). Thx for the catch!

2019-06-01

Aaron Lun (02:22:39): > cool new dependencies badge. Is it counting all dependencies, down to R base packages?

Aaron Lun (02:23:32): > scran’s pretty intense. 93 deps.

Aaron Lun (02:24:39): > woah, scdd is 127

Aaron Lun (02:25:06): > netSmooth is 143!

Aaron Lun (02:25:23): > CATALYST is 181!!

Martin Morgan (10:28:44): > You can read the dependencies with > > views <- "[https://bioconductor.org/packages/devel/bioc/VIEWS](https://bioconductor.org/packages/devel/bioc/VIEWS)" > dcf = read.dcf(url(views), c("Package", "Maintainer", "dependencyCount")) > > The motivation for the badges was that bigger is not really better – code re-use is one thing but excessive dependencies make for very fragile code. > > I did this for a density plot and rug, with Bioc-maintained packages in red > > tbl = as_tibble(dcf) %>% > mutate( > Maintainer = gsub("[[:space:]]+", " ", Maintainer), > dependencyCount = as.integer(dependencyCount), > Core = startsWith(Maintainer, "Bioconductor Package Maintainer") > ) > tbl1 <- arrange(tbl, Core) > ggplot(tbl1, aes(dependencyCount)) + > geom_density() + > geom_rug(color = tbl1$Core + 1) >

2019-06-04

Shweta Gopal (13:57:36): > @Shweta Gopal has joined the channel

Vince Carey (13:57:36): > @Vince Carey has joined the channel

2019-06-10

Simina Boca (12:35:03): > @Simina Boca has joined the channel

Simina Boca (12:35:47): > Hello!:wave:

Simina Boca (12:36:20): > Stupid question here: Is there a way to get log files? We’re having a bit of trouble reproducing an error in a vignettehttps://bioconductor.org/checkResults/3.10/bioc-LATEST/swfdr/malbec1-buildsrc.html

Martin Morgan (14:32:56): > there isn’t anything more than what is on the page; usually problems reproducing errors come from using different versions of software – the devel builder will use the devel version of Bioconductor, with all packages current, so in R 3-6BiocManager::install(version="3.10"); BiocManager::valid()

Simina Boca (14:51:52): > Ah, all right

Simina Boca (14:52:08): > Will ask the contributor who did the update to check the R version

Simina Boca (14:52:29): > Just that the page says “See swfdrQ.log for more info,” so we were wondering if log files were available

Martin Morgan (15:43:52): > This seems like a problem introduced by BiocStyle; I opened an issuehttps://github.com/Bioconductor/BiocStyle/issues/63

Simina Boca (16:05:55): > :pray:= thank you

2019-06-11

FelixErnst (09:35:39): > @FelixErnst has joined the channel

2019-06-26

Junhao Li (13:26:37): > @Junhao Li has joined the channel

2019-08-17

Aaron Lun (02:19:51): > > Dear Package Maintainer, > > > > We would like to bring to your attention that your package is failing to check on our devel build machines: > Ah. I’m clearly not important enough for a personalized email.

Aaron Lun (02:20:56): > Not too far from “Dear Professor Sir/Madam”

Aaron Lun (02:21:35): > “We would like to invite you to the 500th International Conference on Potato Peeling, to be held in Dubai, etc., etc.”

2019-08-26

Aaron Lun (11:46:31): > Urk. Looks like I’ve got myself into a spot of bother with SingleCellExperiment and scRNAseq. I updated the name of the function as discussed in#singlecellexperiment, but SingleCellExperiment doesn’t build on the system because the old scRNAseq is still attempting to get the old function name. I had also updated scRNAseq but I presume it’s waiting for SingleCellExperiment to build properly - catch 22.

Aaron Lun (12:44:29): > @Lori Shepherdor maybe@Hervé Pagès; any thoughts? I guess this doesn’t happen often so maybe it’s just a matter of forcing a new install of SingleCellExperiment to get the build system happy again?

Hervé Pagès (12:50:27): > Not sure what the problem is. Both SingleCellExperiment and scRNAseq are green on the latest build reports:https://bioconductor.org/checkResults/3.10/bioc-LATEST/SingleCellExperiment/andhttps://bioconductor.org/checkResults/3.10/data-experiment-LATEST/scRNAseq/:thinking_face:

Aaron Lun (12:51:00): > hm! Very interesting. Was getting install failures fromBiocManager::install.

Aaron Lun (12:51:35): > And I swear that just an hour ago the build status for SCE was ERROR.

Hervé Pagès (12:53:47): > Right, maybe. But we have a new report everyday y’know. Sometimes it’s just a matter of waiting a couple of extra build cycles, especially since the data-experiment builds run only every other day.

Aaron Lun (12:54:06): > I thought we only got a new report every 2 days.

Hervé Pagès (12:55:02): > Every other day for data-experiment, every day for software. Seehttps://bioconductor.org/checkResults/

Aaron Lun (12:59:08): > Okay. It seems that the build system handles these Suggest/Depends circularities sanely.

Hervé Pagès (13:11:50): > Yep. The key feature is that both software and data-experiment builds use the same R and both builds install the packages to build/check directly from git. This avoids the chicken-egg situation where software package A depends/suggests on data-experiment package B but B can’t be installed because it didn’t propagate yet (because it’s in ERROR), and B also suggests/depends on A but A can’t be installed either because it didn’t propagate yet (because it’s also in ERROR).:face_with_head_bandage:

Aaron Lun (13:13:18): > Cool

2019-09-28

Aaron Lun (21:12:43): > http://bioconductor.org/packages/stats/bioc/batchelor/is giving me a 403 access forbidden error. This doesn’t affect my other packages.

Aaron Lun (21:12:56): > Didn’t think the download rates were so bad that they were unviewable…

2019-09-29

FelixErnst (15:05:30) (in thread): > I think it also affects other packages@Lori Shepherd

Lori Shepherd (15:15:31): > I’ll talk to@Hervé Pagèsabout this thanks for letting us know

2019-09-30

Hervé Pagès (01:31:41): > @Aaron Lun@Lori ShepherdI think I found and addressed the problem. Hopefully things should go back to normal next time the download stats are generated (next Tuesday).

Aaron Lun (01:58:45): > thanks herve

Mike Jiang (13:20:55): > @Mike Jiang has joined the channel

Mike Jiang (13:28:05): > This may be asked/documented somewhere else: How often does Bioc BBS update the system libraries,e.g. libxml2? The reason I asked is because we want to release a flow tool as a shared library to be installed system-wise so that both R packages and non-R programs can link to it. And I wonder how feasible it is for the existing Bioc BBS settings (considering this lib may needs to be updated frequently)?

Mike Jiang (13:44:13) (in thread): > ping@Hervé Pagès@Valerie Obenchain

Hervé Pagès (14:22:12) (in thread): > Hi Mike, the short answer is that in general we don’t control the exact versions of the system libraries we have on the build machines. For example on the Linux builders we use whatever libraries are part of the Ubuntu 18.04 distribution. This is currently: > > biocbuild@malbec1:~$ dpkg-query -s libxml2-dev > Package: libxml2-dev > Status: install ok installed > Priority: optional > Section: libdevel > Installed-Size: 3476 > Maintainer: Ubuntu Developers <ubuntu-devel-discuss@lists.ubuntu.com> > Architecture: amd64 > Multi-Arch: same > Source: libxml2 > Version: 2.9.4+dfsg1-6.1ubuntu1.2 > Depends: libicu-dev, libxml2 (= 2.9.4+dfsg1-6.1ubuntu1.2) > Suggests: pkg-config > Description: Development files for the GNOME XML library > XML is a metalanguage to let you design your own markup language. > A regular markup language defines a way to describe information in > a certain class of documents (eg HTML). XML lets you define your > own customized markup languages for many classes of document. It > can do this because it's written in SGML, the international standard > metalanguage for markup languages. > . > Install this package if you wish to develop your own programs using > the GNOME XML library. > Homepage:[http://xmlsoft.org](http://xmlsoft.org)Original-Maintainer: Debian XML/SGML Group <debian-xml-sgml-pkgs@lists.alioth.debian.org> > > We deliberately stick to that because that’s what 99.9% of Ubuntu 18.04 users will also have on their system. Note that we try to keep our systems up-to-date with the security updates and other software updates (by runningsudo apt-get upgradeabout once a month) so some libraries might get updated sometimes as part of that process. (Valerie has left the core team back in February so please ping@Lori Shepherdinstead for build related subjects. Thanks!)

Mike Jiang (14:27:29) (in thread): > Looks like this won’t work for our case given the once-per-month update frequency for system-wise libs. Thanks a lot for the quick response.

Hervé Pagès (14:40:30) (in thread): > I’m not sure I completely understand your concerns. Every time the Debian or Ubuntu folks provide an update for a system library, AFAIK that doesn’t break the hundreds of executables and other shared libraries that are linked to it. That’s because the new library is supposed to bebinary compatiblewith the previous one (akaABI compatible). Why would that be any different for your flow shared library? And if you don’t trust the ABI compatibility thing you still have the option of making a static library instead of a shared one.

Mike Jiang (14:46:23) (in thread): > You said BBS only upgrades libraries once a month, which I understood as it won’t pick up the changes we may push to Debian/ubuntu every few days,right?

Hervé Pagès (14:59:11) (in thread): > Oh I see. I misunderstood sorry. I thought you wanted to link your flow shared library to other system libraries and you had concerns about how often those other system libraries might be updated. I’m not familiar with the process of contributing to Debian/Ubuntu but note that we use LTS Ubuntu releases on the build machines. The Ubuntu folks are very conservative about changes to LTS releases so these are very stable. I don’t think they would let you push changes to your flow shared lib every few days to an LTS release. My understanding is that this kind of development goes to “Debian unstable” or “Debian testing” first but we’ll never use that on our build machines.

Hervé Pagès (15:02:20) (in thread): > In other words, we won’t pick up the changes you may push to Debian/Ubuntu every few days, even if we were runningsudo apt-get upgradeevery day. And Bioconductor users won’t either.

Mike Jiang (16:01:23) (in thread): > Thanks for the clarification

2019-10-01

Lukas Weber (17:39:52): > @Lukas Weber has joined the channel

Lukas Weber (17:58:14): > Hi, not sure if this is the right channel for this, but I wanted to check about build reports for the packageflowViz. Some of our packages (diffcytandCATALYST) depend onflowViz, I think via chained dependencies fromflowWorkspaceandFlowSOM. > > I rememberflowVizwent through some major updates a while back, and is working again now. However it still has aWARNINGin the Linux build, due to some minor issues in the.Rdfiles. This isn’t creating any problems in the Bioconductor builds, but is creating errors in Travis CI, where warnings are treated as errors by default (and setting it to ignore warnings doesn’t seem to work for dependency packages). Just wanted to check if this is a known issue / planned to be fixed for the next release, and/or whether we can do anything to help. Thanks!

Hervé Pagès (18:27:19): > Hi Lukas, I’m not sure I understand how minor issues in flowViz’s.Rdfiles can affect Travis CI’s build/check of diffcyt or CATALYST. In any case that’s something maybe you want to bring to the attention of the flowViz maintainers, e.g. by opening an issue herehttps://github.com/RGLab/flowViz/issuesOr maybe it reveals a problem with Travis CI itself, don’t know, I don’t use it.

Lukas Weber (18:46:14): > Ok, thank you! I’ll open an issue on GitHub

2019-10-05

Aaron Lun (03:42:30): > ****(Context: playing around with some possibilities for more stable python integration.)****When the build system creates Mac/Windows binaries, how are the hard-coded paths in the shared libraries specified in a portable manner? Is the architecture of these systems so stereotypical that we can just have a single hard-coded path that covers everything?

2019-10-07

Hervé Pagès (13:21:43): > This is really controlled by the package authors. AFAIK packages with difficult system requirements (e.g. Rgraphviz, Rhtslib in BioC devel) have opted for providing Windows binaries that are statically linked in order to free the end users from having to install any system requirements on their end. The situation on Mac is a little bit more complicated: I think that many packages with difficult system requirements are using dynamic linking there (e.g. xps) so they expect the end-user to have the system libraries at specific locations (even though I think this can be mitigated via some-rpathtrickery). But like on Windows, using static linking on Mac is a great option for achieving maximum portability.

2019-10-08

Aaron Lun (20:49:50): > @Hervé PagèsI tagged you on an issue that could do with some of your expertise.

2019-10-09

Aaron Lun (11:38:18): > Looks like our downloads for 2019 disappeared.

2019-10-10

Hervé Pagès (06:56:42): > The AMI we use to compute the stats (stats.bioconductor.org) was restarted on Monday and that interrupted a script that was processing the 2019 stats. We should have complete stats again on Friday.:crossed_fingers:

Lori Shepherd (17:25:02): > My bad:grimacing:sorry everyone

2019-10-11

Hervé Pagès (13:17:55): > Full stats (including stats for 2019) are back:https://bioconductor.org/packages/stats/bioc/

2019-10-17

Aaron Lun (19:45:40): > Is it possible to do a search over all BioC packages to figure out who is using a particular function?

Aaron Lun (19:46:14): > Just in devel… realized I had a wrong name for a function, and I want to fix it before the release, but that may or may not break people’s code.

Shian Su (19:48:35): > Do you really need to do it over all? Should be sufficient to look through reverse dependencies.

Aaron Lun (19:48:45): > Yeah, I don’t want to even do that.

Aaron Lun (19:49:04): > still about 20 packages forscater.

Shian Su (19:50:03): > You can probably cook up a scraper with BiocPkgTools.

Martin Morgan (22:42:00): > It would not be too expensive togit clone --depth 1the reverse dependencies; I don’t think there’s another way at the moment…

Aaron Lun (23:38:55): > Turns out a Github search is pretty good.

Aaron Lun (23:40:51): > I think we discussed this already, but a read-only mirror (with issues and PRs disabled) for all BioC packages would be pretty handy.

2019-10-18

Hervé Pagès (12:18:05): > Problem with the GitHub search is that it assumes all reverse deps are on GitHub which is not necessarily the case. We had a read-only mirror in the past but turned out that the level of confusion it generated outweighted by far its benefits. Personally I have clones of all BioC software packages on my laptop. It’s an expensive operation to clone them all but it’s a one time price to pay. Then I run a script from times to times togit pull themall. This is cheap. I couldn’t really imagine maintaining packages with hundreds of reverse deps without having this kind of setup.

Hervé Pagès (12:24:42): > An alternative to the GitHub search is anrdrr.iosearch (e.g.https://rdrr.io/search?q=uniquifyFeatureNames) but I don’t know how up-to-daterdrr.iois.

Aaron Lun (14:04:49): > @Hervé PagèsI think I’ve identified all affected packages (Spaniel and splatter). I’ve notified the maintainers on their GitHub repositories, but if they don’t respond, could you just edit the affected lines on their behalf to ensure the package builds?

Hervé Pagès (16:11:48): > Where are the edits? Can you point me to the PRs?

Aaron Lun (16:27:19): > There are only two edits to perform, so I didn’t bother making PRs: > -splatter/R/compare.R, lines 82-83:addQCPerCellshould be renamed toaddPerCellQC, andaddQCPerFeatureshould be renamed toaddPerFeatureQC. > -Spaniel/R/readData.R, line 123:addQCPerCellshould be renamed toaddPerCellQC.

Hervé Pagès (17:27:43): > ok will do. But it’s too late for today’s builds (they already started). Note that we also need to bump the version of these packages and make them require scater >= 1.13.27. We should also make sure we canR CMD INSTALL/build/checkbefore committing/pushing. Always takes longer than one might think. Note that you still have an occurrence ofaddQCPerCellin your vignette.

Hervé Pagès (17:46:41): > Done. Tell them to make sure to resync their GitHub repo ASAP.

Aaron Lun (17:47:12): > Thanks, will do.

2019-10-29

Kevin Rue-Albrecht (17:33:07): > @Kevin Rue-Albrecht has left the channel

2019-11-08

Alan O’C (08:20:21): > @Alan O’C has joined the channel

2019-11-20

Mike Jiang (15:49:14): > Does Bioc windows build machine have preinstalled protobuf?(like cranhttps://github.com/rwinlib/protobuf). If so, what version?where to locate? If not, can we consider to have one? Since I am having trouble to build the latest protobuf 3.10 with Rtools 3.5

Mike Jiang (15:54:34): > and which version of Rtools is bioc using?

Hervé Pagès (17:58:08): > Hi Mike, everything we’ve installed on our Windows builders is documented herehttps://github.com/Bioconductor/BBS/blob/master/Doc/Prepare-Windows-Server-2012-HOWTO.TXTYes we have protobuf there, seehttps://github.com/Bioconductor/BBS/blob/ae5a6222e06719e728212d809c3e22c64f4a0cca/Doc/Prepare-Windows-Server-2012-HOWTO.TXT#L774-L790It was installed a long time ago (in Nov 2016 according to the timestamp). Don’t know what version it is though (couldn’t find any indication of this:face_with_raised_eyebrow:). We have Rtools 3.4.0.1962 on tokay2. Probably a little outdated but last time I heard from Jeroen Ooms (the new Rtools maintainer) he was still discussing with the R core people the possibility of a major update of the toolchain for R 4.0. So we’re waiting to know more before we update on tokay2.

Peter Hickey (18:41:47) (in thread): > Re R tools 4.0https://twitter.com/henrikbengtsson/status/1196700481642328064 - Attachment (twitter): Attachment > Awesome - it looks like @opencpu‘s Rtools 4.0.0 is starting to make its way into CRAN. We just got: > > 1. win-builder ’R-devel_gcc8’: “with an alternative toolchain based on gcc 8.0 from Rtools40 (experimental)” > > 2. CRAN checks on the above (https://cran.r-project.org/web/checks/check_flavors.html#r-devel-windows-ix86_x86_64-gcc8) > > #rstats

Mike Jiang (19:27:30) (in thread): > The one you have is probably outdated (protobuf 2.6.0), I was requested by@Martin Morgana while ago to update the bundled src in RProtobufLib package, so I am now having 3.10.0, but I couldn’t get the windows binary (cross-compiled from linux) works properly seehttps://github.com/RGLab/RProtoBufLib/issues/8

Mike Jiang (19:35:06) (in thread): > Basically compilers in Rtools3.5 use another C++ exception model (dwarf/seh) and not compatible with mingw32 from linux (which uses sjlj) . Since you mentioned bioc is still using rtools3.4, I will downgrade mine to 3.4 to see if it fixes the issue

2019-11-25

Mike Jiang (14:15:15): > @Hervé PagèsI managed to build the latest protobuf (3.10) with MYSYS+mingw, but the binary size exceeds the limits

Mike Jiang (14:15:31): > > remote: Error: file larger than 5 Mb. > remote: > remote: File name: 'src/win/lib/i386/libprotobuf-lite.a' > remote: File size: 7.7 Mb >

Mike Jiang (14:16:34): > I prefer to ship the binaries withRProtobufLib(similar to whatRhdf5libdoes)

Mike Jiang (14:17:34): > But it seems to me Rhdf5lib was able to go through the size limits check regardless of its size

Hervé Pagès (14:17:46): > What’s the benefit?

Mike Jiang (14:18:54): > so that we can update it on our own without asking bioc to install it system-wise everytime

Hervé Pagès (14:19:55): > But this seems to happen every 3 years right?

Mike Jiang (14:26:02): > Could be more frequent

Hervé Pagès (14:29:20): > The best scenario would be to have the protobuf source (or a subset of it) included in RProtobufLib and have libprotobuf-lite.a to build as part of theR CMD INSTALL RProtobufLibprocess. How hard would that be? This is what Rhtslib does. If that’s not feasible for RProtobufLib then you’ll have to make the case for an exception to the file size limit with@Martin Morganand@Nitesh Turaga. I don’t control this.

Nitesh Turaga (14:29:32): > @Nitesh Turaga has joined the channel

Mike Jiang (14:32:56): > protobuf needs MSYS environment in order to run itsconfigurescript, I don’t know if that is available throughR CMD INSTALLcontext

Martin Morgan (14:36:37) (in thread): > probably this has been discussed so sorry for the redundancy but is there a reason not to usehttps://cran.r-project.org/web/packages/RProtoBuf/index.html?

Mike Jiang (14:40:30) (in thread): > It’s just a R wrapper, doesn’t ship c++ libs

Mike Jiang (14:43:36) (in thread): > bioc pulls the binary version ofRProtoBuffrom CRAN, right?

Martin Morgan (14:44:00) (in thread): > yes the Windows & Mac binaries get pulled from CRAN, like ‘most’:wink:users would…

Mike Jiang (14:47:57) (in thread): > So bioc probably doesn’t have libprotobuf installed system-wise, which is why we need to ship it inRProtobufLib

Mike Jiang (14:52:40) (in thread): > Because the flow tools uses protobuf c++ APIs

Martin Morgan (14:55:28) (in thread): > (sorry, seem to be having problems getting the right thread here). It sounds like the best solution is to follow CRAN and use a system-wide installation? Sorry if@Hervé Pagèsand I are talking at cross purposes…

Mike Jiang (14:58:25) (in thread): > I am ok with that. (just thought it would be less maintenance work for you guys if I do it inRhdf5libway)

Mike Jiang (15:01:18) (in thread): > also, it would be more consistent with nix platforms, where we already bundled the pb and everything else just simply compile and link againstRProtobufLibinstead of relying on the system libs

Mike Jiang (15:02:57) (in thread): > since system libs are problematic for nix users when they try to install flow tools from bioc

Hervé Pagès (15:08:02) (in thread): > Note that on Windows, you can bypass the configure step because all you want to achieve is haveR CMD INSTALLsucceed onourWindows build machines (tokay1 & tokay2). FWIW I’m taking advantage of this in Rhtslib where I use a Makefile that is customized for the tokays so no need to runconfigure. If you go that route then maybe you will no longer need the MSYS environment? The only drawback about this approach is that Windows users can no longer install the package from source (unless they have things configured like on the tokays of course). But to me the benefits outweigh by far this little inconvenience (and so far nobody has complained about this for Rhtslib).

Hervé Pagès (15:17:34) (in thread): > Anyway, let us know what you decide. If making an exception to the file size limit is not an option and if having libprotobuf-lite.a build as part ofR CMD INSTALL RProtobufLibdoesn’t work for you then I’ll be happy to install the latest protobuf on our Windows builders.

Mike Jiang (15:31:24) (in thread): > I’ll look intoRhtsliband let you know. Thanks a lot!

Mike Jiang (19:52:14) (in thread): > Ended up with the old approach : install pb on server

Mike Jiang (19:52:41) (in thread): > Could you please help with that?

Mike Jiang (19:52:56) (in thread): > > ## Download protobuf Windows binary from[http://rglab.github.io/binaries/](http://rglab.github.io/binaries/)## Extract all the files to C:\protobuf. > > ## Set environment variables LIB_PROTOBUF to C:/protobuf > >

2019-11-26

Hervé Pagès (13:34:44) (in thread): > will do

Hervé Pagès (18:10:44) (in thread): > Hi Mike, the new protobuf .zip file athttp://rglab.github.io/binaries/(protobuf-3.10.0.zip file) doesn’t seem to contain binaries for the 2 archs: 32-bit and 64-bit. FWIW the previous zip file was a bi-arch binary i.e. it had i386 and x64 subfolders in the top folder. If we want to support RProtoBufLib for 32-bit and 64-bit Windows we need to put a bi-arch binary of protobuf on the Windows builders.

Mike Jiang (18:18:05) (in thread): > Yes, it does. see underprotobuf/lib/

Mike Jiang (18:19:47) (in thread): > sinceincludeis identical for both arch, I only split thelibfolder into biarch

Hervé Pagès (18:22:00) (in thread): > Ah ok, I missed that. As long as yourMakevars.winorMakefile.winfiles know about this layout everything should be fine then. I extracted it in the usual place (C:).

Mike Jiang (18:22:53) (in thread): > Great, Thanks a lot!

2019-12-02

Mike Jiang (14:01:52): > Let me know if this channel is the right place to post this kind of issue.tokay2server doesn’t seem to haveRcppParallelinstalled properly, which is causing all the flowtools failing. Here is the error page forcytolibhttp://bioconductor.org/checkResults/3.11/bioc-LATEST/cytolib/tokay2-install.html, and here is the relevant message

Mike Jiang (14:01:55): > > Error: .onLoad failed in loadNamespace() for 'RcppParallel', details: > call: library.dynam("RcppParallel", pkgname, libname) > error: DLL 'RcppParallel' not found: maybe not installed for this architecture? >

Lori Shepherd (14:12:20): > Thank you we will investigate

Lori Shepherd (15:08:57): > There seems to be a very recent version of the binary available. I reinstalled it manual to see if there were any issues and it installed cleanly. I’ll do some additional testing tomorrow morning and keep an eye on the build report tomorrow

2019-12-03

Lori Shepherd (13:46:53): > Looks like this install worked - all green for cytolib today

Lori Shepherd (13:47:02): > http://bioconductor.org/checkResults/devel/bioc-LATEST/cytolib/

Dan Bunis (15:36:01): > @Dan Bunis has joined the channel

Dan Bunis (15:40:58): > Does anyone know a good resource explaining how to build R from source? CRAN isn’t cutting it / seems outdated & my googling is not leading anywhere useful.

Dan Bunis (15:41:38): > I’m hoping to troubleshoot my submitted package’s (dittoSeq) build fails.

Vince Carey (15:51:54): > @Dan Buniswhat platform are you building on?

Dan Bunis (15:52:14): > right sorry, Mac.

Vince Carey (15:54:16): > I used to do this regularly. But the benefits waned relative to the effort of getting all the runtimes together. Are you sure you need this to diagnose the build failure? Is this a newly submitted package?

Dan Bunis (15:57:15): > Newishhttps://github.com/Bioconductor/Contributions/issues/1302

Vince Carey (15:58:08): > Yes, I found it. I will see if I can reproduce the problem. I don’t think building R from source will address the issues noted.

Dan Bunis (15:59:12): > :pray:thank you!

Vince Carey (16:02:08): > This page has been crucial for building R from source if you want to explore that:https://mac.r-project.org/tools/… but I don’t know how up-to-date it is. Posting to thesupport.bioconductor.orgmight be appropriate for more pointers, or there is a mac SIG for R that is probably best – a low traffic mailing list, seer-project.orgMailing lists node.

Vince Carey (16:02:17): > But let’s hope this is not necessary.

Lukas Weber (16:05:26): > There is also a useful page on setting up R compiler tools for MacOS here, not sure if this is also useful for this issue:https://thecoatlessprofessor.com/programming/cpp/r-compiler-tools-for-rcpp-on-macos/

Lori Shepherd (16:06:20): > @Dan BunisFWIW - The RcppParallel on tokay2 should be corrected now as discussed above so you should get a build for that the next time you submit a push - I’ll try and look into the Cairo installation on our end too - I seem to remember Cairo being troublesome in the past too -

Dan Bunis (16:14:23): > @Lori Shepherdthanks! Cairo’s the likely culprit IMO for the Mac check issue. (What I was planning to do was to troubleshoot by seeing if I could recreate the issue with a +/- Cairo build.)

Vince Carey (16:15:39): > when you run capabilities() is the entry for Cairo TRUE?

Vince Carey (16:15:51): > I have had some setbacks in that space but I don’t recall how they were resolved

Dan Bunis (16:23:45) (in thread): > TRUE on my machine.

Vince Carey (16:27:14) (in thread): > I am pretty sure I used brew to get cairo set properly … > > %vjcair> brew info cairo > cairo: stable 1.16.0 (bottled), HEAD > Vector graphics library with cross-device output support[https://cairographics.org/](https://cairographics.org/)Not installed > From:[https://github.com/Homebrew/homebrew-core/blob/master/Formula/cairo.rb](https://github.com/Homebrew/homebrew-core/blob/master/Formula/cairo.rb)==> Dependencies > Build: pkg-config ✘ > Required: fontconfig ✔, freetype ✘, glib ✘, libpng ✔, lzo ✘, pixman ✘ > ==> Options > --HEAD > Install HEAD version > ==> Analytics > install: 76,744 (30 days), 233,875 (90 days), 966,264 (365 days) > install_on_request: 6,060 (30 days), 19,834 (90 days), 80,530 (365 days) > build_error: 0 (30 days) >

Dan Bunis (16:31:43) (in thread): > I’ll look into brew too. I unfortunately need to shift my attention back to some other projects for a while, but thanks a lot for your suggestions!

Shian Su (20:25:55): > So vignettes in my package aren’t building.https://master.bioconductor.org/checkResults/3.10/bioc-LATEST/CellBench/malbec1-buildsrc.html

Shian Su (20:26:21): > Not sure what to do, they used to work and no changes were made, they still build on my local machine. Were there any changes to the build machines recently?

Martin Morgan (22:34:26): > The LaTeX error is apparently from BiocStyle and ‘we’ are working on ithttps://stat.ethz.ch/pipermail/bioc-devel/2019-December/015862.html

2019-12-04

Kevin Rue-Albrecht (15:22:09): > @Kevin Rue-Albrecht has joined the channel

Kevin Rue-Albrecht (15:22:43) (in thread): > Hem.. forget for my email on bioc-devel then:grimacing:

2019-12-05

Laurent Gatto (12:28:30): > @Laurent Gatto has joined the channel

2019-12-08

Aaron Lun (23:21:30): > scater’s vignette is timing out on windows. Haven’t much idea what’s going wrong, but if I had to say, I would guess that it’s stalling atZeiselBrainData()

2019-12-09

Hervé Pagès (13:44:12): > Running the code in the vignettes interactively (source(overview.R, echo=TRUE)) on tokay2 works fine (takes about 4 min, slowest step wasplotHighestExprs()which took about 2 min to generate the plot). HoweverR CMD build scaterseems to take forever so I stopped it after about 25-30 min. Problem seems to be happening in the rmarkdown/knitr/BiocStyle stack. I found this in the log file: > > --- re-building 'overview.Rmd' using rmarkdown > Loading required package: SingleCellExperiment > Loading required package: SummarizedExperiment > Loading required package: GenomicRanges > Loading required package: stats4 > Loading required package: BiocGenerics > Loading required package: parallel > > Attaching package: 'BiocGenerics' > > The following objects are masked from 'package:parallel': > > clusterApply, clusterApplyLB, clusterCall, clusterEvalQ, > clusterExport, clusterMap, parApply, parCapply, parLapply, > parLapplyLB, parRapply, parSapply, parSapplyLB > > The following objects are masked from 'package:stats': > > IQR, mad, sd, var, xtabs > > The following objects are masked from 'package:base': > > Filter, Find, Map, Position, Reduce, anyDuplicated, append, > as.data.frame, basename, cbind, colnames, dirname, do.call, > duplicated, eval, evalq, get, grep, grepl, intersect, is.unsorted, > lapply, mapply, match, mget, order, paste, pmax, pmax.int, pmin, > pmin.int, rank, rbind, rownames, sapply, setdiff, sort, table, > tapply, union, unique, unsplit, which, which.max, which.min > > Loading required package: S4Vectors > > Attaching package: 'S4Vectors' > > The following object is masked from 'package:base': > > expand.grid > > Loading required package: IRanges > > Attaching package: 'IRanges' > > The following object is masked from 'package:grDevices': > > windows > > Loading required package: GenomeInfoDb > Loading required package: Biobase > Welcome to Bioconductor > > Vignettes contain introductory material; view with > 'browseVignettes()'. To cite Bioconductor, see > 'citation("Biobase")', and for packages 'citation("pkgname")'. > > Loading required package: DelayedArray > Loading required package: matrixStats > > Attaching package: 'matrixStats' > > The following objects are masked from 'package:Biobase': > > anyMissing, rowMedians > > Loading required package: BiocParallel > > Attaching package: 'DelayedArray' > > The following objects are masked from 'package:matrixStats': > > colMaxs, colMins, colRanges, rowMaxs, rowMins, rowRanges > > The following objects are masked from 'package:base': > > aperm, apply, rowsum > > snapshotDate(): 2019-10-22 > see ?scRNAseq and browseVignettes('scRNAseq') for documentation > loading from cache > see ?scRNAseq and browseVignettes('scRNAseq') for documentation > loading from cache > see ?scRNAseq and browseVignettes('scRNAseq') for documentation > loading from cache > Loading required package: ggplot2 > Invalid Parameter - /figure-html > Warning in shell(paste(c(cmd, args), collapse = " ")) : > 'convert "overview_files/figure-html/unnamed-chunk-5-1.png" -trim "overview_files/figure-html/unnamed-chunk-5-1.png"' execution failed with error code 4 > Invalid Parameter - /figure-html > Warning in shell(paste(c(cmd, args), collapse = " ")) : > 'convert "overview_files/figure-html/plot-pdata-pct-exprs-controls-1.png" -trim "overview_files/figure-html/plot-pdata-pct-exprs-controls-1.png"' execution failed with error code 4 >

Hervé Pagès (13:44:12): > This warning is a known issue and has already been discussed herehttps://stat.ethz.ch/pipermail/bioc-devel/2019-October/015635.html(and I don’t know if it is related to the timeout – could it be that failing to crop the images causes a timeout later e.g. maybe because the images are so big that it takes forever to embed them in the HTML document or something like that?) Anyway I only realize now how scary this call toconvertis given that on our Windows machines this is callingC:\Windows\system32\convertwhich is a Microsoft utility for converting a file-system from FAT to NTFS (https://docs.microsoft.com/en-us/windows-server/administration/windows-commands/convert) Not good! (Glad the build system runs as a normal user with no admin privileges:fearful:) - Attachment (docs.microsoft.com): convert > Windows Commands topic for ******** -

Aaron Lun (15:34:49): > Hm. What do you recommend?

Hervé Pagès (18:24:20): > Not sure. IIUC theconvertwarning on Windows appeared at some point (a few months ago) after some change happened in knitr and/or BiocStyle. I don’t know if it has been formally reported to the interested parties. I’ll check and, if not, I’ll report. However I’m not 100% convinced that this is what’s causing the time out. I see that there is a Windows binary available for scater in release:scater_1.14.0.zipThis means that at the time we made the release there was no time out. Then only a couple of days after the release scater got modified and its version got bumped in release and devel but the Windows binary never got generated which suggests that the time out is related to these changes (unless it’s caused by some other change that happened around that time in one of the deps). Hard to tell.

Aaron Lun (18:25:53): > :+1:

2019-12-10

Camille BONAMY (11:56:55): > @Camille BONAMY has joined the channel

2019-12-11

Christine Choirat (12:07:20): > @Christine Choirat has joined the channel

Hervé Pagès (21:49:26): > The‘convert …’ execution failed with error code 4warning now reported:https://github.com/yihui/knitr/issues/1785In the mean time we’ll install ImageMagick on the Windows builders and see if that helps with scater’s TIMEOUT.

2019-12-12

Aaron Lun (00:34:35): > :+1:

2019-12-18

Hervé Pagès (03:02:35): > @Aaron LunTheconvert.execommand from ImageMagick is now on tokay2. The good news is that all the‘convert …’ execution failed with error code 4warnings are now gone (this will reflect on today’s build report). The bad news is thatR CMD build scateris still timing out. Can someone with access to a Windows machine try to reproduce this?

Aaron Lun (03:03:05): > Thanks Herve. Do you know where it’s timing out at?

Hervé Pagès (03:13:14): > Hard to know, I can’ttail -ffiles that are being written to by a process on Windows so it’s hard to monitor progress ofR CMD buildon this platform. However, as I reported earlier, I can source the code in the vignette (source(overview.R, echo=TRUE)) and it completes in about 4 min. So I suspect something stalls after that during the conversion to HTML. Maybe if you changed the output toBiocStyle::pdf_documentinstead ofBiocStyle::html_documentwe would learn something?

Hervé Pagès (13:22:36): > No we won’t learn anything because you’ll run into thishttps://github.com/Bioconductor/BiocStyle/issues/68. Anyway, some more troubleshooting indicates that the rendering engineBiocStyle::html_document()doesn’t like theplot-highestchunk: > > > rmarkdown::render("overview.Rmd", BiocStyle::html_document()) > ... > ... > label: unnamed-chunk-10 > |.......................... | 37% > ordinary text without R code > > |........................... | 38% > label: unnamed-chunk-11 > |............................ | 40% > ordinary text without R code > > |............................. | 41% > label: plot-highest (with options) > List of 2 > $ fig.asp : num 1 > $ fig.wide: logi TRUE > > and that’s how far we get. However, when replacingBiocStyle::html_document()withBiocStyle::pdf_document(), the chunk gets evaluated with no problem: > > ... > ... > |........................... | 38% > label: unnamed-chunk-11 > |............................ | 40% > ordinary text without R code > > |............................. | 41% > label: plot-highest (with options) > List of 2 > $ fig.asp : num 1 > $ fig.wide: logi TRUE > > PDFCROP 1.38, 2012/11/02 - Copyright (c) 2002-2012 by Heiko Oberdiek. > ==> 1 page written on `overview_files/figure-latex/plot-highest-1.pdf'. > |.............................. | 42% > ordinary text without R code > > |............................... | 44% > ... > ... (keeps going) > > Next thing I can try is update Pandoc to the latest version. If that doesn’t solve the problem then I’ll be in a position to say that the problem is reproducible (granted you have access to a Windows machine with the latest Pandoc installed) and not due to some configuration issue with our Windows builders.

Aaron Lun (13:23:48): > Thanks@Hervé Pagès. I can just… delete that chunk. It would be fair to say that the idea behind that plot was largely misguided.

Hervé Pagès (13:25:08): > Up to you but let me check the status of Pandoc first.

Aaron Lun (13:26:11): > k

Hervé Pagès (13:30:28): > Oh, interestingly, after more than 20 min running a new emptypngfile showed up inscater\vignettes\overview_files\figure-html: > > PS C:\Users\biocbuild\bbs-3.11-bioc\meat\scater\vignettes\overview_files\figure-html> ls > > > Directory: C:\Users\biocbuild\bbs-3.11-bioc\meat\scater\vignettes\overview_files\figure-htm > > > Mode LastWriteTime Length Name > ---- ------------- ------ ---- > -a--- 12/18/2019 1:22 PM 0 plot-highest-1.png > -a--- 12/18/2019 1:00 PM 86159 plot-pdata-pct-exprs-controls-1.png > -a--- 12/18/2019 1:00 PM 174061 unnamed-chunk-5-1.png > > 20 min to generate a 0-bytepng!

Aaron Lun (13:32:58): > ah windows.

Aaron Lun (13:33:13): > you can see it’s really trying hard.

Hervé Pagès (18:00:30): > Pandoc updated to the latest version (2.9) on the 2 Windows builders. scater still timing out.

2019-12-19

Shubham Gupta (20:38:00): > @Shubham Gupta has joined the channel

Shubham Gupta (20:38:38): > I face a WARNING on Bioconductor build. It says > > Checking R Version dependency... > * WARNING: Update R version dependency from 3.6 to 4.0. > > The latest version I see is 3.6.2 on the R project website. Does anyone know what is causing this warning?http://bioconductor.org/spb_reports/DIAlignR_buildreport_20191219202412.html

Shubham Gupta (20:38:59): > This warning is withR CMD BiocCheck

Shubham Gupta (20:39:38): > I do see that in build_report it uses****RVersion****: 4.0Is there a way to tell bioconductor to test it on lesser version?

Aaron Lun (20:40:03): > Why do you need the version dependency?

Shubham Gupta (20:41:45): > Because when I useR CMD buildIt automatocally adds dependency of >= 3.5 due to some data-structure I am using

Shubham Gupta (20:43:31): > NB: this package now depends on R (>= 3.5.0) > WARNING: Added dependency on R >= 3.5.0 because serialized objects in serialize/load version 3 cannot be read in older versions of R. File(s) containing such objects:

Shubham Gupta (20:44:01): > I think it is because of.RDataobject I am using.

Aaron Lun (20:44:38): > Yes, that would be why.

Aaron Lun (20:45:56): > Well, if you must have Rdata files in your package (and there are definitely better ways, depending on the situation), then the solution is to bump it up to 4.0. Which is what Bioconductor will be using for BioC-devel builds anyway, and that’s what will be with the next release. If you’re concerned about people using it between now and the next release, just have a branch floating around that skips the version requirement.

Aaron Lun (20:46:53): > Bioconductor’s release branch always works with the latest R release. This makes life so much easier for developers, and avoids users getting inconsistent behavior from mismatches between BioC/R versions.

Aaron Lun (20:47:11): > Case in point; the change insample()behavior last year.

Shubham Gupta (20:48:05): > Latest R release is3.6.2, this is way ahead in future. Anyway, having another branch seem to be a good solution.

Aaron Lun (20:48:43): > That’s because new submissions land in the BioC-devel branch.

Aaron Lun (20:51:55): > So the next R release will be 4.0.0, and that’s what will be used with the next BioC release.

Hervé Pagès (23:26:31): > @Shubham GuptaYou’ve checked the box for > > I have read the Bioconductor Package Submission > instructions. My package is consistent with the Bioconductor Package Guidelines. > > which means that you’ve read the “Bioconductor Package Submission instructions”. They say: > > The system will perform these steps using the 'devel' version of Bioconductor, on three platforms (Linux, Mac OS X, and Windows). After these steps are complete, a link to a build report will be appended to the new package issue. Avoid surprises by running these checks on your own computer, under the 'devel' version of Bioconductor, before submitting your package. > > Full version here:https://bioconductor.org/developers/package-submission/You’ve also checked the box where you confirm that you’re familiar with the ‘devel’ branch for new packages and features so there should be no surprise here. Another highly recommended reading is our package guidelines:https://bioconductor.org/developers/package-guidelines/Further questions are better asked in the submission issue tracker where the reviewer assigned to you or another core team member should be able to assist you. > Thanks.

2019-12-20

Shubham Gupta (11:49:10): > I think it has to do withsave(rdaFile, version = 2). I saved my rda files with version = 2 and now I don’t need to have R version dependency

2019-12-21

Aaron Lun (03:39:51): > Download stats broken again.

2019-12-22

Hervé Pagès (14:46:39): > Looks like thestats.bioconductor.orgAMI was restarted again.

2019-12-23

Hervé Pagès (01:05:21): > Not much to do at this point. The download stats should fix themselves by Tuesday.

Lori Shepherd (07:38:36): > Sorry about the trouble this has caused. I’, making better notes in the documents for when updating this instance so we can hopefully avoid this in the future. My apologies.

2020-01-07

Nitesh Turaga (09:18:42): > @Nitesh Turaga has left the channel

Andrew McDavid (23:30:11): > @Andrew McDavid has joined the channel

2020-02-05

Robert Ivánek (02:16:00): > @Robert Ivánek has joined the channel

2020-02-14

Nicholas Cooley (16:19:37): > @Nicholas Cooley has joined the channel

2020-02-19

Nicholas Cooley (10:51:55): > I have what might be a naive question; every once in a while when i push a change to a package, even though the version number has been advanced, it doesn’t seem to trigger a new build, am I doing something incorrectly? Is this a known issue?

Lori Shepherd (11:08:37): > when submitting a new package or for an accepted package

Nicholas Cooley (11:12:49): > a new package

Nicholas Cooley (11:13:01): > err, it’s in the development process now

Nicholas Cooley (11:13:18): > https://github.com/Bioconductor/Contributions/issues/1369

Lori Shepherd (11:25:10): > if the webhook is activated it should trigger a new build when there is a version bump - it is not expected behavior to not trigger a build esp. since you are receive at least some of the version bump triggered messages - It could be an internet connectivity issue, I know about an hour ago I wasn’t able to comment on github - I’ll try and keep an eye on it - the next time it happens please let us know and we can try and investigate some more

Lori Shepherd (11:26:21): > as luck would have it - I happen to be your reviewer too - so if you just want to comment on the issue that you didnt get a build triggered the next time this happens

Nicholas Cooley (11:30:28): > thanks!

2020-03-07

Aaron Lun (01:47:14): > @Martin MorganLooks like basilisk.utils didn’t make it into the build system? It was one of basilisk’s dependencies, for various reasons.

Aaron Lun (02:03:00): > To provide some background; basilisk needs to share code between itsconfigureand its run-time functions. However, I can’t put that code in basilisk functions, because they won’t be available at the time of basilisk’s installation to be run duringconfigure. I didn’t want to copy code, so instead I made a new basilisk.utils package that provides the relevant functionality for bothconfigureand basilisk’s run-time.

Aaron Lun (02:16:39): > If that makes sense, here is the URL:https://github.com/LTLA/basilisk.utils

Martin Morgan (10:52:56): > I will add this for builds on Sunday night

Aaron Lun (12:34:29): > :+1:

2020-03-09

Shubham Gupta (14:31:53): > I am facing an issue with Bioconductor / R build system. I use the commandR CMD buildand this have different behavior for a functionksmoothfromstatspackage in Windows. In linux, everything is fine. > My package name is DIAlignR. The tests related toksmoothandloessfunctions fail withi386but succeed onx64.DIAlignR.Rcheck/tests_i386/testthat.Rout.failfails butDIAlignR.Rcheck/tests_x64/testthat.Routsucceeds. > The web link ishttps://www.bioconductor.org/checkResults/3.11/bioc-LATEST/DIAlignR/tokay2-checksrc.htmlI have printed the output numbers from the tests. Seems like the computation is different for these base_package functions ini386vsx64? Why so? > The link for the Linux build ishttps://www.bioconductor.org/checkResults/3.11/bioc-LATEST/DIAlignR/malbec2-checksrc.htmlThis is successful withDIAlignR.Rcheck/tests/testthat.Rout

Aaron Lun (14:32:44): > IIRC this stems from the use of x87 instructions on i386, which uses 80-bit extended precision floating points for internal calculations.

Aaron Lun (14:33:04): > Well, that’s assuming your code doesn’t have any other bugs.

Shubham Gupta (14:35:46): > Yeah. I kind of know what calculation is resulting the output I see, but can’t imagine that this is due to i386 precision. > In summary, it should take average of next three number but it taking average of next two. Considering that everything is ok onx64andLinux, I would blame i386

Shubham Gupta (14:36:34): > Butloessis a very common function in most of the smoothing and can’t believe that it is not reproducible.

Marcel Ramos Pérez (14:41:24): > @Marcel Ramos Pérez has joined the channel

Nicholas Cooley (15:01:15): > Hi, sorry if this is a bit of a naive question, but I have a submitted package that’s passed it’s build checks, and I’m under the assumption that the next step is a check from a reviewer, but i haven’t heard anything in a while, do I need to do something to trigger that next step? > > My build report:http://bioconductor.org/spb_reports/SynExtend_buildreport_20200219133006.html

Shubham Gupta (15:05:37): > It took me 35 days to get a reply

Shubham Gupta (15:06:29): > The Bioconductor website says that you should receive an answer within 2 months, so if it hasn’t been two months, I would suggest to wait.

Nicholas Cooley (15:06:49): > ok, just wanted to make sure i was actually at the “wait patiently” phase, and hadn’t missed something

Shubham Gupta (15:07:19): > PS: I submitted by package during Christmas, so you may get feedback earlier.

Shubham Gupta (15:12:54): > Is there a way to know if R is using 32-bit windows or 64-bit windows?

Aaron Lun (15:38:29): > Technically yes, with.Machine$sizeof.pointer(4 on 32-bit, 8 on 64-bit). But it is probably better to change your tests to avoid issues with double-precision values very close to a threshold. I’ve done this myself a few times, no big deal.

Shubham Gupta (18:44:31): > Thanks@Aaron Lun. I tested my code with a slight change in param and it does affect the results significantly, I think because in underlying C code there is a filter to pick number of points it want to average on. With slight change in parameter/ precision can leave behind or include boundary point. Writing better test will solve this:slightly_smiling_face:

2020-03-10

Simina Boca (00:28:48): > You can also doSys.info()

Simina Boca (00:29:24): - File (PNG): image.png

Aaron Lun (00:30:11): > Yes, well, mymachineentry is"x86_64", which goes to show how variable it can be.

Simina Boca (00:37:31): > Mine is the same

Simina Boca (00:38:07): > But yeah, if you want to include this in an in/then test, you may need to know what the possible options can be (hopefully they all grep 64?)

Hervé Pagès (03:03:10): > I’m confused. Isn’t the double type following the IEEE 754 standard i.e. is 8 bytes whatever the arch (32-bit or 64-bit) ?

Aaron Lun (03:05:44): > yeah, but x87 uses 80 bits for internal calculations because it doesn’t have access to SSE2 instructions that preserve precision on 64-bit machines.

Aaron Lun (03:06:25): > or something like that. I don’t really remember, but it was some fiddly stuff with the instruction set.

Hervé Pagès (03:11:38): > I don’t seem to be able to find anything definitive about this. Any idea where I should look?

Aaron Lun (03:13:00): > I don’t know much beyond the usual, i.e., wikipedia

Aaron Lun (03:13:19): > https://en.wikipedia.org/wiki/Extended_precision#Need_for_the_80-bit_format - Attachment: Extended precision > Extended precision refers to floating point number formats that provide greater precision than the basic floating point formats. Extended precision formats support a basic format by minimizing roundoff and overflow errors in intermediate values of expressions on the base format. In contrast to extended precision, arbitrary-precision arithmetic refers to implementations of much larger numeric types (with a storage count that usually is not a power of two) using special software (or, rarely, hardware).

Aaron Lun (03:15:22): > I can’t say with 100% certainty that’s the cause of the differences in precision between 32- and 64-bit windows. But it’s the best explanation I have.

Hervé Pagès (03:20:42): > Thanks for the link. My understanding is that extended precision != double precision though. The former is a mess while the latter is well specified, especially in C99.

Hervé Pagès (03:24:09): > I believe we enter extended precision territory when we use something like double double.@Shubham GuptaIs your code relying on double double?

Aaron Lun (03:28:04): > well, that’s the thing. the extended precision is being used as an intermediate in the double-precision calculations.

Aaron Lun (03:28:31): > the link provides an example with exponentiation, but it probably applies to other operations as well.

Aaron Lun (03:32:20): > So even if you write C(++) code that only usesdouble, it’s possible that the machine code is operating with 80-bit doubles under the hood.

Aaron Lun (03:32:39): > Check out the-ffloat-storeoption athttps://gcc.gnu.org/onlinedocs/gcc-3.2/gcc/Optimize-Options.html - Attachment (gcc.gnu.org): Using the GNU Compiler Collection (GCC) > Using the GNU Compiler Collection (GCC)

Aaron Lun (03:35:44): > Or a more extended discussion athttps://gcc.gnu.org/wiki/FloatingPointMath

Hervé Pagès (03:57:29): > Thx for the pointers. Very technical stuff! Too bad R doesn’t use the-msse2flag by default on 32-bit Windows. All Intel procs have had SSE2 for the last 20 years!

Lori Shepherd (09:32:23) (in thread): > I’m working on your review now and should have something back in the next day or two. Sorry for the delay

Lori Shepherd (09:37:59): > The Bioconductor Release 3.11 (tentative) Schedule has been posted. Please see the following for important deadlineshttp://bioconductor.org/developers/release-schedule/. We encourage all maintainers to be active in fixing any package ERRORs in release 3.10 and 3.11 ASAP. Cheers.

Aaron Lun (11:48:07) (in thread): > Probably because some ancient machine Ripley has running somewhere with NT 3.1 or something.

Shubham Gupta (12:51:56) (in thread): > No. I do not specify double double. But I thinki386uses 32 bit andx64uses 64 bit. In the underlying C code ofksmoothfunction from stats package, they use some threshold for selecting points over which some calculation is done. I think this precision can changes number of points and eventually the final result.

Shubham Gupta (12:59:19) (in thread): > This is thelink for C-source code. I am not sure if 32-bit or 64-bit precision could affect underlying calculation

Hervé Pagès (13:08:46) (in thread): > Forget that and see the links@Aaron Lunprovided. I’m still surprised you get such different results on 32-bit Windows compared to 64-bit. I mean: 0.5450333 vs 1.0989811 ! (This is the 1st value displayed right below the “ksmooth output, kernel = box” line.) That smells like numerical instability to me, in which case there is no reason a priori to trust the result you get on 64-bit Windows and Linux either.

Aaron Lun (13:10:02) (in thread): > It’s probably some value that just got bumped above/below a threshold on one machine compared to the other, and that triggered a cascade of other decisions that yield different results.

Aaron Lun (13:10:19) (in thread): > Happens all the time with clustering.

Shubham Gupta (13:10:27) (in thread): > I think culprit could beint krn = asInteger(skrn);Do you know where to find the definition ofasInteger? > > The reason is numerical instability oftestorC code? I think it is because of the C code.

Hervé Pagès (13:14:11) (in thread): > Of the underlying algo itself. Hard to know where exactly. Numerical instability is a property of the algo itself and is more pronounced when the algo involves many iterations. No matter the programming language or the amount of precision.

Aaron Lun (13:21:14) (in thread): > I would say that, as numeric imprecision goes, this is nothing. It’s when you start playing with t-SNE and UMAP that propagation really starts, seehttps://github.com/jlmelville/uwot/issues/46for an example. Watch a difference in the 16th significant figure completely change your results!

Hervé Pagès (13:28:58) (in thread): > doesn’t sound good:worried:

2020-03-12

Mike Jiang (14:13:12): > Windows binary is not availablehttp://bioconductor.org/packages/3.11/bioc/html/flowClust.htmleven though the build report says it succeedhttp://bioconductor.org/checkResults/3.11/bioc-LATEST/flowClust/tokay2-buildsrc.html(there is check error but I guess it is not relevant ?) - Attachment (Bioconductor): flowClust (development version) > Robust model-based clustering using a t-mixture model with Box-Cox transformation. Note: users should have GSL installed. Windows users: ‘consult the README file available in the inst directory of the source distribution for necessary configuration instructions’.

Lori Shepherd (14:17:41): > There is a check error and it is relevant. We don’t propagate broken packages. That includes install/build/check

Mike Jiang (14:27:39): > I see, just pushed the fix.

Mike Jiang (14:28:18): > Also, I didn’t see the build report for mac, any reason for that?

Lori Shepherd (14:33:06): > https://stat.ethz.ch/pipermail/bioc-devel/2020-January/016057.htmlWe have not yet fixed the issues on our end. We hope to have the mac builder running again sooon

Simina Boca (15:08:22): > Any updates on this error on tokay2? “! LaTeX Error: Environment cslreferences undefined.”

Martin Morgan (15:16:03) (in thread): > context?

Simina Boca (15:58:12) (in thread): > We have it in this packagehttp://bioconductor.org/checkResults/devel/bioc-LATEST/swfdr/tokay2-buildsrc.htmlbut@Lori Shepherdhas mentioned that this was probably on the Bioconductor end. It’s present in a few other packages as well.

Lori Shepherd (15:59:06) (in thread): > We just updated the devel windows builder this afternoon. Let’s see if this persists after the update.

Simina Boca (15:59:33) (in thread): > :thumbsup:

Simina Boca (15:59:45) (in thread): > Just making sure I wasn’t missing anything on my end!

2020-03-13

Sean Davis (07:06:22): > @Sean Davis has joined the channel

2020-03-15

Simina Boca (15:17:31) (in thread): > My package at least still has an error:disappointed:

Lori Shepherd (20:12:14) (in thread): > Okay. Thanks. We will look into it this week

2020-03-16

Lori Shepherd (12:41:10) (in thread): > @Hervé Pagèspointed me to this closed issue, Once the version hits CRAN it should clear up automaticallyhttps://github.com/rstudio/rmarkdown/issues/1649

Lori Shepherd (13:33:42) (in thread): > we will temporarily also install the github version manually so that it clears up in the meantime

2020-03-17

Aaron Lun (14:46:30): > Has something changed with 32-bit windows compilation in the latest R-devel? I’m getting errors in BiocNeighbors where there were none before; these likely arise from differences in compilation settings between CRAN and Bioconductor.

Hervé Pagès (23:39:50): > Don’t know what could cause this. Lori can confirm but I think she updated R the usual way on the Windows builder i.e. by grabbing the latest R devel build fromhttps://cran.r-project.org/bin/windows/base/rdevel.htmlThis should give us the same compilation settings that they use on CRAN.

2020-03-18

Lori Shepherd (08:47:49): > Yes I grabbed the latest and followed the standard procedure

Steffen Neumann (16:57:53): > @Steffen Neumann has joined the channel

Steffen Neumann (17:31:48): > Hi, I am trying to help Egon Willighagen with his problem, see[Bioc-devel] help with Win10 TLS problem?. Istokay1runnign rtools3.5 or already rtools 4.0 ? Which SSL version is used on tokay1 ? Is the next release using rtools 4.0, and in that case,@Jeroen Giliscould we have bridgeDB included inhttps://bioconductor.org/checkResults/3.11/bioc-testing-LATEST/? Minimal reproducible example which fails with TLS error on windows 10:download.file("https://zenodo.org/record/3611238/files/PubChemLite_14Jan2020_tier0.csv?download=1", "dummy.csv")

Martin Morgan (17:36:28): > My understanding is that tokay1 is running rtools 3.5. CRAN hasn’t announced that they will use rtools 4.0 for the next release (and we are too close to the release to change now…), and we can’t build bioc packages that are incompatible with CRAN. Probably you want to tag@Hervé Pagèsabout addition to the bioc-testing builds. I think it’s better to follow up with the MRE on the place where the conversation was initiated – the bioc-devel thread.

Hervé Pagès (17:55:23): > Yeah, no Rtools 4.0 yet for our daily builds. See my answer on bioc-devel about the BridgeDbR’s issue on Windows.

2020-03-19

Aaron Lun (04:05:01) (in thread): > Hm. I wonder why that’s happening, then.

Aaron Lun (04:13:36) (in thread): > scran’s error is even more bizarre. I can’t even imagine howhttp://bioconductor.org/checkResults/devel/bioc-LATEST/scran/tokay2-checksrc.htmlcomes to be.

Aaron Lun (04:16:09) (in thread): > Guess that’s another one for the “give-up on windows” bin.

Mike Smith (08:59:31): > @Mike Smith has joined the channel

Daniela Cassol (12:29:50): > @Daniela Cassol has joined the channel

2020-03-20

Aaron Lun (15:20:28): > I remember GitHub Actions being discussed here, so here’s an example run of an Action to automatically take screenshots of iSEE instances based on the vignettes in theiSEEpackage repository:https://github.com/iSEE/iSEE/pull/390/checks?check_run_id=522787913

Aaron Lun (15:20:49): > Further details in#isee

Hervé Pagès (16:01:50): > As discussed during yesterday’s meeting the build report now provides a link to the R settings used on the build machines:https://bioconductor.org/checkResults/3.11/bioc-LATEST/messina/malbec2-checksrc.html

2020-03-24

Jake Wagner (17:11:16): > @Jake Wagner has joined the channel

2020-03-26

Mike Jiang (14:32:01): > dockerhub showsbioconductor/bioconductor_docker:RELEASE_3_10was successfully built a day ago, but after Idocker pull bioconductor/bioconductor_docker:RELEASE_3_10, and started the container, > > root@fbb5d60a2571:/# R --version > R version 3.6.2 (2019-12-12) -- "Dark and Stormy Night" > > which is behind the current bioc night builds3.6.3 (2020-02-29) -- "Holding the WindsockSo I couldn’t really use the docker image to reproduce the current error reported in bioc 3.10, the same thing applies todevelbranch according to the last try from@Jake Wagner. > Not sure if this has been asked here before, Is it true that bioc docker image doesn’t necessarily use the exact same environment as bioc build server?

Lori Shepherd (14:50:14): > Did you already have a version of the docker images pulled down previously?

Mike Jiang (14:54:30): > I don’t think so > > L$ docker images > REPOSITORY TAG IMAGE ID CREATED SIZE > bioconductor/bioconductor_docker RELEASE_3_10 494d2f47f0da 25 hours ago 3.61GB > wjiang2/gs-to-flowjo latest ece854683d7d 5 days ago 36.6MB > alpine latest 961769676411 7 months ago 5.58MB > ubuntu latest 4c108a37151f 9 months ago 64.2MB > gstack/ubuntu-trusty latest bf8936bcff95 20 months ago 473MB >

Sean Davis (15:02:53): > It looks like the image digest for release is different from what you have:https://hub.docker.com/r/bioconductor/bioconductor_docker/tags

Mike Jiang (15:08:00): > It was freshly pulled this morning, not sure what else I can do other than > > docker pull bioconductor/bioconductor_docker:RELEASE_3_10 > RELEASE_3_10: Pulling from bioconductor/bioconductor_docker > Digest: sha256:9810c72e6c09997e64209084c8fa5ea6708e8ca8b655f780c83e0fcc34fa7ae5 > Status: Image is up to date for bioconductor/bioconductor_docker:RELEASE_3_10 >

Jake Wagner (15:19:24): > I see the same problem as Mike and am getting the same image digest after a fresh pull (same as Mike, not same as what dockerhub shows)

Hervé Pagès (16:48:05): > Asking on the#containerschannel will probably increase your chances of getting help on this.

Nitesh Turaga (20:13:43): > @Nitesh Turaga has joined the channel

2020-03-30

Minoo (11:22:01): > @Minoo has joined the channel

2020-04-01

Mike Jiang (13:57:08): > I shipped some boost source code under package src folder (i.e. cytolib/src/boost/libs/system/*.c), butR CMD buildcompiles and generates .o files for these source code(only forsrc/boostsubdir, but not the source undersrc/) and automatically include them into the finaltar.gzfile for linux/mac, which is causing CRAN to complain about these .o files. Does anyone know how to stop R from generate these binary objects when build the source tar ball?

Kevin Rue-Albrecht (14:34:00): > @Kevin Rue-Albrecht has left the channel

Hervé Pagès (15:06:06): > My understanding is thatR CMD buildonly takes care of cleaningsrc/by default. For any other cleaning you need to provide a cleanup script. Seehttps://cran.r-project.org/doc/manuals/r-release/R-exts.html#Configure-and-cleanup - Attachment (cran.r-project.org): Writing R Extensions > Writing R Extensions

Mike Jiang (15:23:03) (in thread): > That was it!Thanks for the quick solution!

Nicholas Cooley (18:00:13): > Can someone help me understand the R version dependency warning I am getting from the build servers? > > If I include R >= 4.0 in the description, I get a warning to remove the dependency, even though I have compressed RData files. If I remove the dependency, a dependency of R >= 3.5 gets added and the warning remains?

Dan Bunis (18:07:22): > What I did, which may not be best practice so separate question perhaps (?), was to save my data files withsave(…, version = 2). When saved this way, R does not flag a warning nor does it add the R >= 3.5 dependency in building my package.

Nicholas Cooley (18:10:03): > But I feel like I added the R >= 4.0 at some point in the submission process because of a separate warning earlier. I stepped away from this for a while because it’s just hard to juggle everything, so I might be misremembering.

Dan Bunis (18:13:27): > All I can say is that data files were definitely the reason I’d added an R dependency myself:man-shrugging:

Hervé Pagès (18:23:39): > Stick to R >= 4.0 which is the right thing to do. The warning you got when you had R >= 4.0 was spurious and should have been ignored. (Was discussed yesterday in the#generalchannel, seehttps://community-bioc.slack.com/archives/C35G93GJH/p1585639082012200). FWIW we’ve updated R to R 4.0 alpha this morning on the builders and this spurious warning should go away. - Attachment: Attachment > Can anyone help out with / give any insights re the following: BiocCheck using R-devel/Bioc3.11 gives me > This is BiocCheck version 1.23.1. BiocCheck is a work in progress. > Output and severity of issues may change. Installing package... > * Checking Package Dependencies... > * Checking if other packages can import this one... > * Checking to see if we understand object initialization... > * Checking for deprecated package usage... > * Checking for remote package usage... > * Checking version number... > * Checking for version number mismatch... > * Checking version number validity... > * Checking R Version dependency... > * WARNING: Update R version dependency from 4.0 to 4.1. > I then changed Depends: R (>= 4.0) to 4.1 in the DESCRIPTION , after which all ran cleanly. But the Bioc server build fails with ERROR: this R is version 4.0.0, package 'muscat' requires R >= 4.1 . Now I could just ignore the error locally… but this is really confusion still. Should BiocCheck be giving this warning?

Nicholas Cooley (18:27:57): > Ohp, I missed that! Thanks!

Nicholas Cooley (18:49:55): > So just to make sure I understand, it is still returning the warning with the dependency readded, and that’s expected, and the warning label will be removed automatically when the builders are updated? or i’ll need to recommit once they’re updated?

Martin Morgan (19:23:20) (in thread): > Actually, BiocCheck was updated with > > commit 8bcf30eb7be938f8ad369b5f2a7cda48510092ac (HEAD -> master, origin/master, origin/HEAD) > Author: lshep <[lori.shepherd@roswellpark.org](mailto:lori.shepherd@roswellpark.org)> > Date: Wed Apr 1 09:37:02 2020 -0400 > > Update R version dependency check > > - should not be default in DESCRIPTION dependencies > - its an R package of course it will depend on R > - caveat is if data is compressed/saved as if there are changes to underlying objects, structures, default object arguments it could invalidate a saved object > > The reasoning being that specifying an R dependency seems redundant (what other language would this run on?) and unnecessary for Bioconductor (because the standard installation procedures don’t allow installation on a version of R that the package is ‘known good’ (i.e., build and check successfully on the build system). And sayingR (>= 4.0)is somehow claiming clairvoyance (is that the right term, ‘to see into the future’?), e.g., packages that relied onstringsAsFactors = FALSEare broken in R-4.0 alpha / Bioconductor devel, even though they promiseR (>= 3.y). > > as@Dan Bunisalludes to, there is unfortunately a warning generated when using data serialized in a way that requires R >= 2.10. But this version of R is ancient, and I think the warning should be removed… (I’ll lobby for that…)

Hervé Pagès (19:30:03): > Just revert the recent changes you made as an attempt to get rid of the original warning. That is: 1) reserialize your data normally (i.e. without specifyingversion=2), and 2) restore dep on R >= 4.0 again. Then bump the version (zpart only) to trigger a new build. You should no longer see the warning but, if you see it, please ignore it.

2020-04-02

Aaron Lun (03:43:51): > Has anything changed recently re. python on the windows build machines? I’m getting - sigh - another set of windows-only build failures forbasilisk.

Aaron Lun (03:50:06): > honestly, who knows what’s going on there. Fingers crossed it’s just some intermittent network error.

2020-04-06

Hervé Pagès (17:49:59): > <!channel>An update on the “Blame Jeroen” builds (https://bioconductor.org/checkResults/3.11/bioc-testing-LATEST/): > 1. These builds are now using the latest Rtools 4.0 (beta 22) and latest R-testing: > > > R version 4.0.0 alpha (Rtools40) (2020-03-31 r78116) -- "Blame Jeroen" > > 2. Here is what theBuiltfield looks like for binary packages build with R-testing + Rtools 4.0: > > Built: R 4.0.0; x86_64-w64-mingw32; 2020-04-06 19:33:53 UTC; windows > > Compare with what it looks like for binary packages built with current R-4.0-alpha + Rtools 35: > > Built: R 4.0.0; i386-w64-mingw32; 2020-01-28 17:57:14 UTC; windows > > The only difference is that for packages built with the new Rtools we seex86_64-w64-mingw32 instead ofi386-w64-mingw32. Very subtle:pensive:3. The “Blame Jeroen” builds are still running every day (this has not changed since I started them in January). > > 4. I’ll keep an eye on tomorrow’s report and notify maintainers of packages that have issues with the new toolchain. > > 5. Not 100% sure yet that the CRAN folks want to switch to the new toolchain before the R 4.0.0 release (they’re still debating it) but we need to be ready to switch just in case. > > Thanks!

Aaron Lun (17:51:22): > haha

Aaron Lun (17:52:15): > I wonder whether this was the trigger for my BiocNeighbors failures. Seems like there’s yet another change in numerical precision for 32-bit windows.

2020-04-07

Steffen Neumann (02:43:59): > I like the “Blame Jeroen” name with a wink, and I think all kudos also go to Jeroen for this ! Would it make sense to have them less-well hidden in the normal build reports ?

2020-04-10

Aaron Lun (13:20:58): > I have to say, running conda viabasiliskon the windows build machine is a bit like flipping a coin. Maybe it installs, maybe it doesn’t, seems like it’s all down to luck.

Martin Morgan (13:43:16): > Probably the cost of new installs? and the solution is to cache somehow?

Aaron Lun (13:44:23): > In fact,basiliskdoes caching by default; but I also test the caching; which involves wiping the cache and repopulating it; and those tests are failing (apparently) stochastically.

Aaron Lun (13:45:10): > damned if I do and damned if I don’t. I don’t mind eating the windows build failures, but it does make me nervous that it’s hiding some problem underneath.

2020-04-11

Mike Jiang (00:05:47): > flowWorkspaceis failing some test cases only on win 32bit, I am planning to simply skip these offending tests on win32 assuming 32bit R is rarely used thus not worth the efforts. Am I missing something on that? ( I figured there must be reasons for bioc to still build binaries for both arch, but not sure why)

Aaron Lun (02:22:16): > I would also be very interested to know that.

Martin Morgan (05:58:54): > flowWorkspaceis also failing in devel unit tests on the new macOS builds. > > CRAN supports 32-bit windows so we do. > > Not infrequently the errors on ‘non-standard’ platforms are pointing to underlying problems, especially in C code, where the different architecture triggers memory access bugs. The bugs are likely problematic on all platforms, just hidden from testing. > > There are occasional questions on the support site where sessionInfo() or other signs show that usersdouse 32-bit windows. Why? I don’t know, maybe they click on the wrong icon on their desktop, maybe they really are on 32-bit machines. When your package doesn’t support 32-bit windows you’ll contribute to people who are confused by the message that ‘flowWorkspace’ (or more cryptically one of its reverse dependencies) ‘is not available in R-x.x’. This makes work for others trying to debug the situation, and tarnishes whatever reputation Bioconductor has for reliable package availability across platforms. > > Common problems are unit tests that expect 64-bit accuracy on 32-bit platforms (bad unit test) or that naively assume excessive memory resources (bad unit test), or… > > On the Windows flowWorkspace devel builds there are a number of ‘warnings’ in the C++ code that could easily contribute to cross-platform instabilities. The notes and warnings also indicate that the documentation needs significant attention. It seems to me like the first step here is to clean up the notes and warnings, then actually tackle the bugs for more robust code.

2020-04-12

Mike Jiang (21:18:58) (in thread): > Thanks for the advice. We will try to do as you recommended. Btw, any tips for debug c code on windows? I can’t seem to step into my c++ code by followinghttps://cran.r-project.org/bin/windows/base/rw-FAQ.html#How-do-I-debug-code-that-I-have-compiled-and-dyn_002eload_002ded_003f - Attachment (cran.r-project.org): R for Windows FAQ > R for Windows FAQ

Mike Jiang (21:21:48) (in thread): > The only thing I didn’t follow was that my R was installed from binary downloaded from CRAN. I recall I used to be able to do it without building R from source with debug mode enabled.

2020-04-13

Pierre-Luc Germain (03:23:42): > @Pierre-Luc Germain has joined the channel

Martin Morgan (04:08:44) (in thread): > I’ve debugged C code using the general instructions in the link you provide, but that was quite a long time ago. As on other platforms it is probably important to make sure the compiler sees flags-g -O0to include debugging symbols and to avoid optimization. On other platforms one might have a file ~/.R/Makevars > > CXXFLAGS="-g -O0" > > but I am not sure whether this is sufficient; perhaps someone else in the community has more experience for developing at this level on Windows?

Mike Jiang (12:26:30) (in thread): > right, That is what I’ve been doing on linux, which works fine. But windows seems to behave differently. I will post it to broader community. Thanks!

Aaron Lun (12:48:34): > Trying to debug BioCNeighbors 32-bit failures now, which stem from a mismatch in behavior between my code and RcppAnnoy (despite the fact that they use the same header-only library). I assume we’re pulling the binaries for 4.0.0 from CRAN - does anyone know the compilation settings used on the CRAN build machines? I’m looking athttps://cran.r-project.org/web/checks/check_flavors.html#r-devel-windows-ix86_x86_64but I’d like to get the actual GCC flags.

Hervé Pagès (14:27:19): > I can’t tell for sure but I believe that they use whatever settings are reported byR --arch i386 CMD config <VAR>(using the official Windows built of R). I realize now that we only display the x64 settings herehttps://bioconductor.org/checkResults/3.11/bioc-LATEST/tokay2-NodeInfo.html(by defaultR CMD config <VAR>displays the compilation settings for 64-bit Windows) and I don’t know how different the settings are for 32-bit Windows. Let me go on tokay2 and check that.

Aaron Lun (14:28:46): > It looks pretty similar based on the echo’d gcc calls.

Hervé Pagès (14:35:19): > > PS C:\Users\biocbuild\bbs-3.11-bioc> R\bin\R --arch i386 CMD config CC > C:/Rtools/mingw_32/bin/gcc > PS C:\Users\biocbuild\bbs-3.11-bioc> R\bin\R --arch i386 CMD config CFLAGS > -O2 -Wall -std=gnu99 -mfpmath=sse -msse2 -mstackrealign > PS C:\Users\biocbuild\bbs-3.11-bioc> R\bin\R --arch i386 CMD config CXX > C:/Rtools/mingw_32/bin/g++ -std=gnu++11 > PS C:\Users\biocbuild\bbs-3.11-bioc> R\bin\R --arch i386 CMD config CXXFLAGS > -O2 -Wall -mfpmath=sse -msse2 -mstackrealign > > Yup, pretty similar indeed. I wouldassumethat’s what they use on CRAN too (that’s the whole point of providing and using the official R built for Windows). What gcc calls you see on the CRAN compilation report for RcppAnnoy?

Aaron Lun (14:37:37): > That’s the thing. I don’t get to see the INSTALL logs.

Hervé Pagès (14:40:07): > Oh right, they only provide the link to the INSTALL log when there is an installation error. This sucks.

Aaron Lun (14:42:52): > History suggests that it is a difference in compilation settings, based on the case ofhttps://stat.ethz.ch/pipermail/bioc-devel/2018-December/014450.html. Was-mfpmath=sse -msse2a recent addition?

Hervé Pagès (14:49:36): > The fact that CXXFLAGS is set to-O2 -Wall -mtune=core2in R 3.6 and toO2 -Wall -mfpmath=sse -msse2 -mstackrealignin R 4.0 suggests that. I don’t know anything about the motivation behind this change though or what the implications are (I didn’t check R’s NEWS, or svn log, or the R-devel mailing list).

Aaron Lun (14:50:48): > I wonder whether it is a simple matter of RcppAnnoy (and other packages) not having been recompiled after the switch to the latest toolchain.

Hervé Pagès (14:56:30): > AFAIK there has been no change to the toolchain yet.

Aaron Lun (14:57:20): > Or whatever the switch was betweenmtune=core2and the new set of flags.

Aaron Lun (14:57:28): > It was in the last month, IIRC.

Hervé Pagès (14:59:44): > The latest version of RcppAnnoy (0.0.16) was published on CRAN on 2020-03-08 and the Windows binaries are in sync with that version so were made after that. So you’re saying that the compilation flags were changed after that? One way to find out if the RcppAnnoy stale binary is what’s breaking BiocNeighbors is to reinstall RcppAnnoy from source on tokay2 and see what happens to BiocNeighbors tomorrow. I’ll do this.

Aaron Lun (15:00:01): > Oh, that would be great, thanks.

Hervé Pagès (15:35:04): > Full INSTALL log for RcppAnnoy on tokay2:

Hervé Pagès (15:35:04): > > > install("RcppAnnoy", type="source") > Bioconductor version 3.11 (BiocManager 1.30.10), R 4.0.0 alpha (2020-03-31 r78116) > Installing package(s) 'RcppAnnoy' > trying URL '[https://cran.rstudio.com/src/contrib/RcppAnnoy_0.0.16.tar.gz](https://cran.rstudio.com/src/contrib/RcppAnnoy_0.0.16.tar.gz)' > Content type 'application/x-gzip' length 441533 bytes (431 KB) > downloaded 431 KB > > * installing **source** package 'RcppAnnoy' ... > **** package 'RcppAnnoy' successfully unpacked and MD5 sums checked > **** using staged installation > **** libs > > ***** arch - i386 > C:/Rtools/mingw_32/bin/g++ -std=gnu++11 -I"C:/Users/BIOCBU~1/BBS-3~1.11-/R/include" -DNDEBUG -I../inst/include/ -I'C:/Users/biocbuild/bbs-3.11-bioc/R/library/Rcpp/include' -I"C:/extsoft/include" -O2 -Wall -mfpmath=sse -msse2 -mstackrealign -c RcppExports.cpp -o RcppExports.o > C:/Rtools/mingw_32/bin/g++ -std=gnu++11 -I"C:/Users/BIOCBU~1/BBS-3~1.11-/R/include" -DNDEBUG -I../inst/include/ -I'C:/Users/biocbuild/bbs-3.11-bioc/R/library/Rcpp/include' -I"C:/extsoft/include" -O2 -Wall -mfpmath=sse -msse2 -mstackrealign -c annoy.cpp -o annoy.o > C:/Rtools/mingw_32/bin/g++ -std=gnu++11 -I"C:/Users/BIOCBU~1/BBS-3~1.11-/R/include" -DNDEBUG -I../inst/include/ -I'C:/Users/biocbuild/bbs-3.11-bioc/R/library/Rcpp/include' -I"C:/extsoft/include" -O2 -Wall -mfpmath=sse -msse2 -mstackrealign -c arch.cpp -o arch.o > In file included from C:/Users/BIOCBU~1/BBS-3~1.11-/R/include/R.h:91:0, > from C:/Users/biocbuild/bbs-3.11-bioc/R/library/Rcpp/include/Rcpp/r/headers.h:71, > from C:/Users/biocbuild/bbs-3.11-bioc/R/library/Rcpp/include/RcppCommon.h:29, > from C:/Users/biocbuild/bbs-3.11-bioc/R/library/Rcpp/include/Rcpp.h:27, > from arch.cpp:3: > C:/Users/BIOCBU~1/BBS-3~1.11-/R/include/R_ext/RS.h:55:0: warning: "ERROR" redefined > #define ERROR ),error(R_problem_buf);} > ^ > In file included from C:/Rtools/mingw_32/i686-w64-mingw32/include/windows.h:71:0, > from ../inst/include/mman.h:14, > from ../inst/include/annoylib.h:46, > from arch.cpp:2: > C:/Rtools/mingw_32/i686-w64-mingw32/include/wingdi.h:75:0: note: this is the location of the previous definition > #define ERROR 0 > ^ > C:/Rtools/mingw_32/bin/gcc -I"C:/Users/BIOCBU~1/BBS-3~1.11-/R/include" -DNDEBUG -I../inst/include/ -I'C:/Users/biocbuild/bbs-3.11-bioc/R/library/Rcpp/include' -I"C:/extsoft/include" -O2 -Wall -std=gnu99 -mfpmath=sse -msse2 -mstackrealign -c init.c -o init.o > C:/Rtools/mingw_32/bin/g++ -shared -s -static-libgcc -o RcppAnnoy.dll tmp.def RcppExports.o annoy.o arch.o init.o -LC:/extsoft/lib/i386 -LC:/extsoft/lib -LC:/Users/BIOCBU~1/BBS-3~1.11-/R/bin/i386 -lR > installing to C:/Users/biocbuild/bbs-3.11-bioc/R/library/00LOCK-RcppAnnoy/00new/RcppAnnoy/libs/i386 > > ***** arch - x64 > C:/Rtools/mingw_64/bin/g++ -std=gnu++11 -I"C:/Users/BIOCBU~1/BBS-3~1.11-/R/include" -DNDEBUG -I../inst/include/ -I'C:/Users/biocbuild/bbs-3.11-bioc/R/library/Rcpp/include' -I"C:/extsoft/include" -O2 -Wall -mfpmath=sse -msse2 -mstackrealign -c RcppExports.cpp -o RcppExports.o > C:/Rtools/mingw_64/bin/g++ -std=gnu++11 -I"C:/Users/BIOCBU~1/BBS-3~1.11-/R/include" -DNDEBUG -I../inst/include/ -I'C:/Users/biocbuild/bbs-3.11-bioc/R/library/Rcpp/include' -I"C:/extsoft/include" -O2 -Wall -mfpmath=sse -msse2 -mstackrealign -c annoy.cpp -o annoy.o > C:/Rtools/mingw_64/bin/g++ -std=gnu++11 -I"C:/Users/BIOCBU~1/BBS-3~1.11-/R/include" -DNDEBUG -I../inst/include/ -I'C:/Users/biocbuild/bbs-3.11-bioc/R/library/Rcpp/include' -I"C:/extsoft/include" -O2 -Wall -mfpmath=sse -msse2 -mstackrealign -c arch.cpp -o arch.o > In file included from C:/Users/BIOCBU~1/BBS-3~1.11-/R/include/R.h:91:0, > from C:/Users/biocbuild/bbs-3.11-bioc/R/library/Rcpp/include/Rcpp/r/headers.h:71, > from C:/Users/biocbuild/bbs-3.11-bioc/R/library/Rcpp/include/RcppCommon.h:29, > from C:/Users/biocbuild/bbs-3.11-bioc/R/library/Rcpp/include/Rcpp.h:27, > from arch.cpp:3: > C:/Users/BIOCBU~1/BBS-3~1.11-/R/include/R_ext/RS.h:55:0: warning: "ERROR" redefined > #define ERROR ),error(R_problem_buf);} > ^ > In file included from C:/Rtools/mingw_64/x86_64-w64-mingw32/include/windows.h:71:0, > from ../inst/include/mman.h:14, > from ../inst/include/annoylib.h:46, > from arch.cpp:2: > C:/Rtools/mingw_64/x86_64-w64-mingw32/include/wingdi.h:75:0: note: this is the location of the previous definition > #define ERROR 0 > ^ > C:/Rtools/mingw_64/bin/gcc -I"C:/Users/BIOCBU~1/BBS-3~1.11-/R/include" -DNDEBUG -I../inst/include/ -I'C:/Users/biocbuild/bbs-3.11-bioc/R/library/Rcpp/include' -I"C:/extsoft/include" -O2 -Wall -std=gnu99 -mfpmath=sse -msse2 -mstackrealign -c init.c -o init.o > C:/Rtools/mingw_64/bin/g++ -shared -s -static-libgcc -o RcppAnnoy.dll tmp.def RcppExports.o annoy.o arch.o init.o -LC:/extsoft/lib/x64 -LC:/extsoft/lib -LC:/Users/BIOCBU~1/BBS-3~1.11-/R/bin/x64 -lR > installing to C:/Users/biocbuild/bbs-3.11-bioc/R/library/00LOCK-RcppAnnoy/00new/RcppAnnoy/libs/x64 > **** R > **** demo > **** inst > **** byte-compile and prepare package for lazy loading > **** help > ***** installing help indices > converting help for package 'RcppAnnoy' > finding HTML links ... done > AnnoyIndex html > RcppAnnoy-package html > getArchictectureStatus html > **** building package indices > **** installing vignettes > **** testing if installed package can be loaded from temporary location > ***** arch - i386 > ***** arch - x64 > **** testing if installed package can be loaded from final location > ***** arch - i386 > ***** arch - x64 > **** testing if installed package keeps a record of temporary installation path > * DONE (RcppAnnoy) > > The downloaded source packages are in > 'C:\Users\biocbuild\AppData\Local\Temp\3\RtmpMZ0Zyr\downloaded_packages' > Old packages: 'foreign', 'lhs', 'nlme', 'xfun' > Update all/some/none? [a/s/n]: >

Hervé Pagès (15:35:04): > After that I gaveR CMD check BiocNeighborsa try but the tests still fail in 32-bit mode.

Aaron Lun (15:36:39): > Is there a-msse2off the end of those calls?

Hervé Pagès (15:39:25): > Are the gcc calls truncated? Oh no, stupid PowerShell terminal. Let me try to copy/paste this again for you.

Aaron Lun (15:41:40): > I’m guessing that it was probably the same as the BiocNeighbors flags.

Aaron Lun (15:41:52): > ARGH. It’s happening all over again.

Hervé Pagès (15:59:47): > I edited the INSTALL log above to show the full gcc commands. Yes the-msse2flag is used (consistent withR CMD config CXXFLAGS).

Aaron Lun (16:59:48): > Looking over it. No differences in build conditions for the exact same header libraries. No differences in macros. I can only say:

Aaron Lun (16:59:49): > ¯*(ツ)*/¯

Aaron Lun (17:00:04): > So I’ve just skipped the offending tests on windows.

Hervé Pagès (18:02:55): > > -- 1. Failure: findAnnoy() behaves correctly on simple inputs (@test-find-annoy. > out$index not identical to ref$index. > 5/5000 mismatches (average diff: 491) > [4024] 551 - 44 == 507 > [4479] 940 - 601 == 339 > [4568] 811 - 24 == 787 > [4780] 923 - 150 == 773 > [4811] 568 - 619 == -51 > > Does it even make sense to support BiocNeighbors on 32-bit Windows if the results are garbage?

Aaron Lun (18:41:29): > It’s “okay”, for some definition of “okay”. This is approximate nearest neighbors, and the problem here is that the approximation is different between 32-bit and 64-bit; this alters the indices of the neighbors. The indices of adjacent neighbors have no obligation to be close together in the input matrix, hence the crazy-looking diffs.

Aaron Lun (18:42:43): > In any case, the offending tests have joined the long and storied set of “tests I have given up debugging on 32-bit windows” with some sweetskip_on_os("windows").

Hervé Pagès (19:00:33): > I see. BTWskip_on_os("windows")will also skip the tests on 64-bit Windows AFAICT. That sounds a little bit drastic.

Aaron Lun (19:01:25): > Yeah. Well. Guess I could look at the pointer size and skip only on 32-bit.

Hervé Pagès (19:01:42): > seriously?

Aaron Lun (19:01:54): > yup.

Aaron Lun (19:02:20): > Well, there’s something in.Machinethat gives the size of the int, so I could use that.

Hervé Pagès (19:03:50): > What aboutis_win32 <- function() .Platform$OS.type == "windows" && .Platform$r_arch == "i386"

2020-04-14

Mike Smith (03:35:20): > Yes, I haveif(.Platform$r_arch != "i386") {}around some of my tests that don’t make sense on 32-bit.

Mike Smith (03:36:50): > Does it make sense to also test for whether it’s Windows, or would issues like this also affect any other 32-bit OS?

Hervé Pagès (04:04:29): > Good question. I guess it depends on the particular issue. I useis_win32()to skip some examples that use too much memory for 32-bit Windows. AFAIK the memory limit is specific to 32-bit Windows and doesn’t affect 32-bit Linux. Even though I have not run a 32-bit Linux distro for at least 12 or 15 years so don’t remember exactly (and at the time, 3 Gb of RAM was a lot of memory anyway and no example would ever use that). Also I don’t know what.Platform$r_archwould look like on a 32-bit Linux distro (it’s set to""on 64-bit Ubuntu).

Mike Jiang (13:21:35): > Just want to hear some opinion here regarding to this issuehttps://github.com/RGLab/cytolib/issues/37

Mike Jiang (13:23:09) (in thread): > I’ve been trying thestatic libroute for the past couple of days, but had some random segfault issues as described in the issue page

Mike Jiang (13:23:36) (in thread): > And I am thinking about a different route , which is documented herehttps://cran.r-project.org/doc/manuals/r-release/R-exts.html#Linking-to-other-packages - Attachment (cran.r-project.org): Writing R Extensions > Writing R Extensions

Mike Jiang (13:26:00) (in thread): > which seems to be feasible (even though not conventional as static lib approach). But since I couldn’t figure out the cause of the segfault cause from static link at the moment. I attempt to stick to shared lib

Mike Jiang (13:33:22) (in thread): > But before I try this, I’d like to hear some advice from BioC experts. I figured there should be some convincing reasons that this approach is not currently widely adopted within Bioc community. (especially likeRhdf5lib,otherwise, it will be obviously more memory efficient than static linking) The doc warns about portability issue, which doesn’t seems to be much of concern to me as long as it covers linux/mac/solaris + gcc/clang

Mike Jiang (13:33:46) (in thread): > ping@Mike Smith@Hervé Pagès

Mike Jiang (13:38:17) (in thread): > BTW, forcytolib, it needs to link toRProtoBufLibandRhdf5lib, if I am to switch to static lib, all the rest offlowtool chains (e.g. flowCore,flowWorkspace, CytoML ..) will need to have their separate copies of these 3 libs, which seems to be such a significant overhead/waste

Hervé Pagès (14:33:53) (in thread): > FWIW I switched from dynamic to static linking for Rhtslib and its clients about 1 year ago:https://github.com/Bioconductor/Rhtslib/commit/db1d8e17ef5b8568fdae3fae0dc701fe2250c952No regrets so far. It seems to be a lot more robust than dynamic linking. IMO this outweighs the small memory footprint reduction that you get with dynamic linking. One thing to keep in mind is that static variables affect the size of a static library e.g. something likestatic double a[1000000];will increase the size of the static library by 8M so you might want to revisit what you do with static variables in cytolib (54M forlibcytolib.ais suspicious). Most of the time you should be able to use dynamic allocation instead.

Mike Jiang (14:59:59) (in thread): > Thanks for the feedback!It definitely helps us to avoid the same hassles you’ve dealt with. > > The size may be caused y header-only libs used by cytolib, e.g. boost ,armadillo , also I am putting many cytolib APIs in the header files as inline functions, which also further contributes to it. I will try to separate them into source files to see if that makes any difference

2020-04-15

Mike Smith (10:58:05) (in thread): > I have also run into problem with dynamic linking on cluster environments, where you can’t necessarily assume that the directory structure will be the same on the execution node compared to where ever the package was built.

Aaron Lun (17:56:11): > Bemusing:http://bioconductor.org/checkResults/devel/bioc-LATEST/basilisk/malbec2-buildsrc.html

Aaron Lun (17:56:23): > basilisk.utils::getLockFileis right there!

Aaron Lun (17:57:07): > What’s going on here?

Martin Morgan (18:07:18) (in thread): > in part, check out the link to the number of installed packages on the tokay2 line in the table at the top ofhttp://bioconductor.org/checkResults/devel/bioc-LATEST/index.html(today the number is 3598) and you’ll see the version of basilisk.utils installed - 0.99.9 – which I think is beforegetLockFilewas introduced…

Aaron Lun (18:08:25) (in thread): > Surely the build system is smart enough to build and check dependencies before attempting to build a package?

Martin Morgan (18:08:48) (in thread): > …and it looks like 0.99.10 built and was successfully pushed today (green LED onhttp://bioconductor.org/checkResults/devel/bioc-LATEST/basilisk.utils/tokay2-buildsrc.html) so tonight’s build should use 0.99.10

Martin Morgan (18:09:00) (in thread): > …although I think there is a big change in the works for the Windows toolchain

Aaron Lun (18:21:44) (in thread): > I’m pretty sure I pushed both of them at the same time.https://bioconductor.org/developers/gitlog/. Seems likebasiliskshould have been able to use the latestbasilisk.utilsfrom the same build cycle.

Hervé Pagès (20:58:12) (in thread): > Yes that’s what normally happens i.e. all software packages are pulled out from git and reinstalled at the beginning of a build run. Except that yesterday something bad happened. Clue: see the the blue stuff in the INSTALL column? Nothing got re-installed! At the root of this disaster is this bug:https://bugs.r-project.org/bugzilla/show_bug.cgi?id=17763The problem should go away on tomorrow’s build report.

Hervé Pagès (21:09:16): > Important build system update: today we’ve switched to Rtools40 + the latest built of R 4.0.0 beta on tokay2. We’ll try to include the new build/check results for tokay2 on tomorrow’s report if tokay2 finishes in time. It started late and it needs to reinstall all the deps from scratch (3000+ packages) so the current build run will take longer than usual. Also we’ll try to propagate the new binaries. We’ve been told by the authorities that all the Windows binaries that are currently in the BioC 3.11 software repo will need to be replaced with the new ones.

Mike Jiang (21:28:14) (in thread): > Is this going to affect the upcoming release as well?

Hervé Pagès (21:41:29) (in thread): > Yep. The switch is real. R 4.0.0 made the switch and BioC 3.11 is based on R 4.0.0 so we need to switch too. I’ve not seen an official announcement on R-devel about the switch yet but hopefully there will be one soon.

Al J Abadi (22:31:27): > @Al J Abadi has joined the channel

2020-04-16

Mike Jiang (00:13:26) (in thread): > ok. I will try to rebuild protobuf system lib (if you haven’t done so)

Hervé Pagès (00:55:22) (in thread): > I’ve not done so, never tried it, no idea how to do that. Sorry.

Constantin Ahlmann-Eltze (03:11:55): > @Constantin Ahlmann-Eltze has joined the channel

Mike Smith (03:29:10) (in thread): > I don’t know if this is a reversion on your part, or something required by Rtools40, but I think we need--force-biarchincluded on the installation step. At the moment I see errors like: > > Error: package 'Rhdf5lib' is not installed for 'arch = i386' > Execution halted >

Hervé Pagès (03:58:28) (in thread): > We don’t have the new devel report yet. The current report (where this Rhdf5lib error appears) is from yesterday. I’m not sure what’s causing the error but it cannot be related to the use of Rtools40 since I updated the toolchain on tokay2 after this report was generated. It could be related to thishttps://community-bioc.slack.com/archives/CEQ04GKEC/p1586998692133800?thread_ts=1586987827.132600&cid=CEQ04GKECWe’re usingR CMD INSTALL --merge-multiarchto install packages on the Windows builders, as always. - Attachment: Attachment > Yes that’s what normally happens i.e. all software packages are pulled out from git and reinstalled at the beginning of a build run. Except that yesterday something bad happened. Clue: see the the blue stuff in the INSTALL column? Nothing got re-installed! At the root of this disaster is this bug: https://bugs.r-project.org/bugzilla/show_bug.cgi?id=17763 The problem should go away on tomorrow’s build report.

Mike Smith (06:57:06) (in thread): > Hopefully that’s it. > > I’m a bit confused asR CMD check --force-multiarch(full command copied from the build report) run in a command line still doesn’t seem to work for me on my own Windows machine. I get that samearch = i386error andRhdf5lib/Rcheck/00install.outcontains the following, which implies that’s true. > > Warning: this package has a non-empty 'configure.win' file, > so building only the main architecture > > However it will check fine if I use the ‘check’ button in Rstudio. > > I notice that on the release build you use the--libraryargument, but not on devel. Would that location already be populated from a previous run, and so the 32-bit install would already be there? > > Should just add that I’m happy to wait a day and see what happens rather than wanting any action.

Hervé Pagès (14:17:40) (in thread): > The new report is not ready yet (and tokay2 is running so late that I don’t think it’ll be ready in time to be in the new report). I went on the build machines and looked at the new CHECK result for Rhdf5lib on tokay2 and everything looks fine. > To clarify: on the Windows builders we useR CMD INSTALL --merge-multiarchto enforce a bi-arch install (otherwise IIRC packages with a configure script only get installed for 64-bit), andR CMD check --force-multiarchto make sure that the examples and unit tests are checked for the 2 archs.

Hervé Pagès (17:17:20): > New build report is online with results for tokay2 (Rtools40):https://bioconductor.org/checkResults/3.11/bioc-LATEST/No results for tokay2 in the BUILD BIN column today (the generation of the new binaries didn’t complete in time). Hopefully we’ll have them tomorrow. Many packages in the “flow stack” are broken because at the bottom of the stack cytolib doesn’t compile yet with the new toolchain.@Mike Jiangis working on it. > Please ignore the hundreds ofpolygon edge not founderrors we see today on machv2 (they are related to theno font could be found for family "Arial"warnings). It’s something on our side. Should go away tomorrow.

2020-04-17

Aaron Lun (00:36:30): > Alright. Back staring at the Windows build failures.

Aaron Lun (00:37:24): > It’s so strange! I’m getting errors for all of my code that uses external C++ libraries.

Aaron Lun (00:37:57): > It’s a compilation issue. It HAS to be a compilation issue. I just don’t see any other explanation.

Aaron Lun (00:50:05): > Aw geez. the 32-bit Windows problems have infectedscranas well. There’s not even any C++ code in that test! URGH.

Aaron Lun (00:58:29): > OMG.

Aaron Lun (00:58:34): > I think I’ve figured it out.

Aaron Lun (01:02:52): > It’s because the C++ code computes distances infloatspace. But the R code that I’m comparing to computes it indoublespace. And, for some reason, the difference in precision is particularly bad on Windows 32-bit, resulting in the minute discrepancies.

Aaron Lun (01:06:13): > I still think the initial errors are due to differences in the compilation of the binaries; but there were also additional errors due to the float/double differences, and I suspect@Hervé Pagèsran into those when we did that little RcppAnnoy reinstallation experiment above.

Aaron Lun (01:26:26): > @Hervé Pagèscould I ask you to do another experiment for me? Could you reinstall RcppHNSW from source on tokay2 and re-check BiocNeighbors? I’d like to see if the current tests continue failing; they would be comparing against equivalently compiled calls into the same library across two packages.

Hervé Pagès (01:55:26): > The Windows builds are so fragile that going on tokay2 right now to do this little experiment would compromise them. I could try tomorrow after they finish. Slack me a reminder here between 11:30 am and 1:30 pm~EST~~PCT~PST tomorrow (yes the window between 2 build runs is very narrow).

Aaron Lun (01:56:29): > Thanks, I’ll try to remember.

Aaron Lun (02:03:15) (in thread): > I realized what the problem is. The function was using some statistics computed previously, where the rowsums were computed in C++. This had slightly different numerical precision from a rowsum computed by R. Looking at R’s C code, this is because it uses along doubleinsum()and friends, which presumably uses 80 bits of precisions on x86 processors. I don’t know why this never manifested before, I would suspect that the new compilation flags exposed this behavior.

Aaron Lun (02:03:41): > Is that PST?

Hervé Pagès (02:31:37): > oops, yes

Charlotte Soneson (05:53:10): > @Charlotte Soneson has joined the channel

Hervé Pagès (13:34:16): > @Aaron LunBuilds are done on tokay2 (the new report will become available in 1h or so) so I went on the machine and re-installed RcppHNSW from source: > > > install("RcppHNSW", type="source") > Bioconductor version 3.11 (BiocManager 1.30.10), R 4.0.0 beta (2020-04-14 r78227) > Installing package(s) 'RcppHNSW' > trying URL '[https://cran.rstudio.com/src/contrib/RcppHNSW_0.2.0.tar.gz](https://cran.rstudio.com/src/contrib/RcppHNSW_0.2.0.tar.gz)' > Content type 'application/x-gzip' length 26095 bytes (25 KB) > downloaded 25 KB > > * installing **source** package 'RcppHNSW' ... > **** package 'RcppHNSW' successfully unpacked and MD5 sums checked > **** using staged installation > **** libs > > ***** arch - i386 > /mingw32/bin/g++ -std=gnu++11 -I"C:/Users/BIOCBU~1/BBS-3~1.11-/R/include" -DNDEBUG -I../inst/include/ -I'C:/Users/biocbuild/bbs-3.11-bioc/R/library/Rcpp/include' -I"C:/extsoft/include" -DNO_MANUAL_VECTORIZATION -DSTRICT_R_HEADERS -O2 -Wall -mfpmath=sse -msse2 -mstackrealign -c RcppExports.cpp -o RcppExports.o > /mingw32/bin/g++ -std=gnu++11 -I"C:/Users/BIOCBU~1/BBS-3~1.11-/R/include" -DNDEBUG -I../inst/include/ -I'C:/Users/biocbuild/bbs-3.11-bioc/R/library/Rcpp/include' -I"C:/extsoft/include" -DNO_MANUAL_VECTORIZATION -DSTRICT_R_HEADERS -O2 -Wall -mfpmath=sse -msse2 -mstackrealign -c hnsw.cpp -o hnsw.o > /mingw32/bin/g++ -shared -s -static-libgcc -o RcppHNSW.dll tmp.def RcppExports.o hnsw.o -LC:/extsoft/lib/i386 -LC:/extsoft/lib -LC:/Users/BIOCBU~1/BBS-3~1.11-/R/bin/i386 -lR > installing to C:/Users/biocbuild/bbs-3.11-bioc/R/library/00LOCK-RcppHNSW/00new/RcppHNSW/libs/i386 > > ***** arch - x64 > /mingw64/bin/g++ -std=gnu++11 -I"C:/Users/BIOCBU~1/BBS-3~1.11-/R/include" -DNDEBUG -I../inst/include/ -I'C:/Users/biocbuild/bbs-3.11-bioc/R/library/Rcpp/include' -I"C:/extsoft/include" -DNO_MANUAL_VECTORIZATION -DSTRICT_R_HEADERS -O2 -Wall -mfpmath=sse -msse2 -mstackrealign -c RcppExports.cpp -o RcppExports.o > /mingw64/bin/g++ -std=gnu++11 -I"C:/Users/BIOCBU~1/BBS-3~1.11-/R/include" -DNDEBUG -I../inst/include/ -I'C:/Users/biocbuild/bbs-3.11-bioc/R/library/Rcpp/include' -I"C:/extsoft/include" -DNO_MANUAL_VECTORIZATION -DSTRICT_R_HEADERS -O2 -Wall -mfpmath=sse -msse2 -mstackrealign -c hnsw.cpp -o hnsw.o > /mingw64/bin/g++ -shared -s -static-libgcc -o RcppHNSW.dll tmp.def RcppExports.o hnsw.o -LC:/extsoft/lib/x64 -LC:/extsoft/lib -LC:/Users/BIOCBU~1/BBS-3~1.11-/R/bin/x64 -lR > installing to C:/Users/biocbuild/bbs-3.11-bioc/R/library/00LOCK-RcppHNSW/00new/RcppHNSW/libs/x64 > **** R > **** inst > **** byte-compile and prepare package for lazy loading > **** help > ***** installing help indices > converting help for package 'RcppHNSW' > finding HTML links ... done > RcppHnsw-package html > hnsw_build html > hnsw_knn html > hnsw_search html > **** building package indices > **** testing if installed package can be loaded from temporary location > ***** arch - i386 > ***** arch - x64 > **** testing if installed package can be loaded from final location > ***** arch - i386 > ***** arch - x64 > **** testing if installed package keeps a record of temporary installation path > * DONE (RcppHNSW) > > The downloaded source packages are in > 'C:\rtools40\tmp\RtmpIf6izW\downloaded_packages' > > Currently runningR CMD check BiocNeighbors_1.5.4.tar.gz…

Aaron Lun (13:34:33): > fingers crossed

Aaron Lun (13:34:35): > thanks.

Aaron Lun (13:34:43): > Probably will still have some errors, but lets’ see what they are.

Aaron Lun (13:48:22): > @Hervé PagèsAny news?

Aaron Lun (13:48:48) (in thread): > okay, great.

Hervé Pagès (13:55:08): > The good news is that the unit tests were successful but there are other problems. They could be related to the fact that I was running R CMD check in the Rtools Bash terminal included with the new Rtools40. I generally don’t use this terminal but I decided to try it instead of the horrible PowerShell terminal because it makes copy/paste easier and it has other interesting features (syntax coloring, use of/instead of\, overall it’s more Unix-like). However, some important components needed byR CMD checkare not on the PATH so I get a bunch of warnings and in the end an error. I’m trying again in the PowerShell…

Aaron Lun (13:57:51): > that’s encouraging.

Hervé Pagès (14:12:58): > R CMD check BiocNeighbors_1.5.4.tar.gzis clean in the PowerShell. Hurrah!

Aaron Lun (14:13:28): > YES

Aaron Lun (14:13:30): > YES

Aaron Lun (14:13:37): > THank god

Aaron Lun (14:14:06): > thanks@Hervé Pagès. I think the problem was in two parts.

Aaron Lun (14:15:05): > The first was the compilation differences between stale CRAN binaries and our re-compilation from source. The second was the fact that the use of SSE on windows resulted in even greater numerical precision differences on 32-bit, which caused additional tests to fail.

Hervé Pagès (14:20:33): > Based on the timestamps herehttps://cran.r-project.org/bin/windows/contrib/4.0/you can see that many of the new Windows binaries are still being rebuilt almost every day. The last binary for RcppHNSW is from yesterday so is more recent than the binary that tokay2 picked up when we started the builds after the update to Rtools40. Hopefully this new binary solves the problem.

Mike Jiang (14:47:28): > Which version of R is built with rtool4 so that it automatically knows the path of rtools (along with its compilers as well as external libs installed by pacman)?

Mike Jiang (14:48:07): > I am using R4-beta, which seems still looks for rtools3?

Hervé Pagès (14:55:32): > We don’t build R, we use the official binaries provided here:https://cran.r-project.org/bin/windows/base/rpatched.htmlIt expects Rtools to be inC:\rtools40which is the default location when you run the Rtools40 installer. So if you use official binaries everything works out of the box.

Mike Jiang (14:57:59): > ok. My R was from official binary, just a few days earlier than bioc

Mike Jiang (14:58:07): > R version 4.0.0 beta (2020-04-09 r78186)

Mike Jiang (14:59:14): > I will update with the latest then.

Hervé Pagès (14:59:15): > I think the switch to Rtools40 happened after that. I don’t understand why they didn’t announce this switch on the R-devel mailing list yet!

Mike Jiang (14:59:49): > Thanks for the reply!

Mike Jiang (15:22:13): > Btw,C:\rtools40\usr\binstill needs to be set inpathenvironment variable, since some package looks for system command (e.g.XMLexpectshto be in the path for its configure script). But the rest seems to be out-of-box, which is nice!

Hervé Pagès (15:37:54): > Oh right! I forgot about the need to prepend some stuff to thePath. YesC:\rtools40\usr\binbut I also like to haveC:\rtools40\mingw32\binandC:\rtools40\mingw64\bineven though it’s not required. Seehttps://github.com/Bioconductor/BBS/blob/master/Doc/Prepare-Windows-Server-2012-HOWTO.md#installation-instructions-for-rtools40for how we set up things on the Windows builders. You’re right, it’s not as out-of-the-box as one would wish.:grin:

Mike Jiang (15:42:52) (in thread): > Maybe we should install packages from source at the moment to avoid the potential incompatibility issues between the old binaries (built with gcc 4.9) and new R(built with gcc8)?

Hervé Pagès (15:56:18) (in thread): > Everything that is inhttps://cran.r-project.org/bin/windows/contrib/4.0/was build with gcc8. The old stuff is in the4.0gcc493folder which is next to the4.0folder (one level up). They started to build binaries with gcc8 months ago and they used to have them in a4.0gcc8folder that they recently renamed4.0. Many important changes and 0 communication:pensive:Anyway, what I meant is that even though they already have gcc8-built binaries for most CRAN packages in the4.0folder, they still seem to be rebuilding a lot of them every day. For the purpose of the Windows builds, we just grab what’s there. It’s not an option for us to install the CRAN deps from source on Windows.

Mike Jiang (16:03:54) (in thread): > I see. Thanks for clarification.

Mike Jiang (21:15:29) (in thread): > @Hervé PagèsI’ve successfully rebuilt protobuf with rtools4, > You can either download it fromhttp://rglab.github.io/binaries/to replace the old one in c:/protobuf , assumingLIB_PROTOBUFenvironment is still present > or you can build it from source (shipped with RProtoBufLib) by followinghttps://github.com/RGLab/RProtoBufLib/blob/master/INSTALL#L14I’ve tested andcytolibnow builds ok with the new pb binary

Hervé Pagès (21:17:56) (in thread): > Sounds good. I’ll try to do this tomorrow after the current builds finish. Thanks!

2020-04-18

Aaron Lun (15:32:13): > Good grief. The stochasticity ofbasiliskfailures on tokay2 is insane. Fix one thing and you get a whole bunch of different unrelated failures.

Hervé Pagès (17:07:04) (in thread): > New Rtools40-built of protobuf is now on tokay2. I tried to install a few packages (RProtoBufLib, cytolib, flowCore, ncdfFlow, and flowWorkspace) and all is fine now. That’ll make a big difference on~today’s~tomorrow’s build report. Thanks@Mike Jiang

2020-04-20

Aaron Lun (02:19:13): > @Hervé PagèsWonder if you could help me out here.http://bioconductor.org/checkResults/devel/bioc-LATEST/basilisk/tokay2-checksrc.htmlindicates a failure but emits no actual error messages. Methinks the entire R process is just dying at some point. If you installed the latestbasilisk(0.99.62) and just rancl <- basiliskStart(NULL), what do you see?

Pierre-Luc Germain (04:15:30): > Dear all, in case anybody has a tip, I’ve been getting a CHECK warning (in windows only) which I really can’t figure out: > > * checking contents of 'data' directory ... WARNING > Output for data("exampleDEAresults", package = "pipeComp"): > Warning messages: > 1: In normalizePath(path.expand(path), winslash, mustWork) : > path[1]="C:\Users\pkgbuild\AppData\Local\basilisk\basilisk\basilisk-0.99.34\anaconda/python.exe": The system cannot find the file specified > ... > > (I really have no clue why python should have anything to do with my package though, except perhaps through some dependencies)… > Is this related to the more general basilisk issues you’ve been discussing, or something else I should continue to try to crack?

Aaron Lun (04:16:11): > What the… that’s my thing!

Aaron Lun (04:16:21): > Well, my thing from 28 versions ago.

Aaron Lun (04:16:55): > What package is this?

Aaron Lun (04:17:21): > Right pipecompo.

Aaron Lun (04:19:50): > I don’t even know what it would do that. Possibly if it were opening separate R processes, e.g., via SnowParam, and those processes were fighting with basilisk processes somehow, where the basilisk processes were generated by some stale R installation somewhere in tokay2.

Pierre-Luc Germain (04:22:34): > thanks (good to know I’m not crazy), so you think it’s not strictly speaking related to the package? Did you fix your error somehow, or it just disappeared from itself randomly?

Aaron Lun (04:24:07): > Well, it can’t possibly be related to your package if you’re not using basilisk. Smells like some interference between builds if tokay2 is building in parallel.

Aaron Lun (04:29:14): > Perhaps that might explain where all my error messages went.

Pierre-Luc Germain (04:32:17): > though I must say it’s the second time I get the message and it’s always at the same step (data dir), which would be unexpected if the message was actually from parallel builds…

Vince Carey (05:16:39): > @Hervé Pagès@Aaron LunIn the most recent report on Tokay, BiocSklearn times out. This is consistent with my experience on local windows 10 machine where the acquisition of the anaconda environment takes a very long time. And if I understand the setup, every time basilisk version changes, the process must repeat for all clients. It would be nice to have a way to factor the acquisition out of the package build process. Users may be willing to pay the price of a long installation, for weeks of pleasant usage, but the package building process may not be able to handle this.

Aaron Lun (11:52:06): > I think this is just a manifestation of the fact that basilisk itself is having trouble building on Windows.

Aaron Lun (11:53:05): > Normally. basilisk would build and install anaconda; then biocsklearn would build and install its environment. Currently, because basilisk builds are failing, biocsklearn is forced to install the entire stack. I’d be surprised if this caused the time-out…. but Windows, so I guess anything could be happening.

Hervé Pagès (13:07:17): > Yes, as you say… Windows! I will re-iterate my very early suggestion that you get your hands dirty on a Windows machine for the purpose of developing/maintaining/troubleshooting basilisk. With Windows and such a tricky package as basilisk, there is NO other way, I’m afraid.

Aaron Lun (13:07:48): > well, that’s the thing.@Vince Careyand I have been hacking at it, and we’re getting clean builds on his Windows 10 machine.

Hervé Pagès (13:12:54): > That’s encouraging. Our Windows builder runs Windows Server 2012. Could this make a difference?

Hervé Pagès (13:13:28): > Or what do you think could explain the difference?

Aaron Lun (13:14:41): > I hypothesize that there are multiple levels to this problem.

Aaron Lun (13:15:42): > The first is that, on Windows,basiliskseems to spawn processes like there’s no tomorrow. Recall that basilisk has an option of either loading python directly into the current R session (i.e., by loading the shared library) or creating a new process and loading Python in that one (e.g., if the current session already has Python loaded).

Aaron Lun (13:16:26): > It seems thatbasiliskon Windows does not even try to do the former, even when it is permitted to do so. IthinkI have fixed this in 0.99.63 but who knows.

Aaron Lun (13:18:49): > Anyway, the propensity of process spawning on Windows is probably the cause of the underlying problem, which is that some of the processes are… getting killed? Or they can’t even spawn? I’m not sure, there aren’t any error messages from the tokay2 check. I assume that R CMD check creates a new slave R process to (i) run the examples and (ii) run the tests; the lack of error messages would be consistent with a ugly kill of those slaves.

Aaron Lun (13:19:27): > In the meantime, 0.99.63 should reduce the number of processes that are getting generated. This may help, but I doubt it.

Vince Carey (13:19:50): > @Hervé PagèsIf you can give me some hints on how to get an evaluation version of Windows Server 2012 running under virtual box I will use it. I downloaded an ISO but I can’t get it to go … maybe the “Server” concept is too alien to me.

Aaron Lun (13:21:13): > @Vince Careyin the meantime, if you have the time, could you just clone the latest version of basilisk (https://github.com/LTLA/basilisk) and run R CMD build/check for me on your VM?

Hervé Pagès (13:25:03): > @Vince CareyNo idea how to do that, sorry.@Aaron Lun”I’m not sure, there aren’t any error messages from the tokay2 check.” But if you were running this locally, there are a lot of things you could do to monitor what’s going on exactly and get some very valuable insight.

Vince Carey (13:42:04): > @Aaron Lunmotivation.Rmd lacks a library(BiocStyle) – needed this to build .63 on windows

Aaron Lun (13:43:39): > Thanks@Vince Carey. Doesn’t seem related to the tokay2 errors, but I’ll throw it in. (Sort of weird that it doesn’t show up anywhere else, but whatever.) Any news on the CHECK?

Vince Carey (13:43:52): > check is running now

Vince Carey (13:44:43): > maybe the build system has a global attachment of BiocStyle?

Aaron Lun (13:46:00): > Doubt it.

Vince Carey (13:49:25): > check is moving smoothly and seems to be in the anaconda installation stage. i get a popup window very briefly indicating the download. i will be out for an hour or so. will get back to you – the laptop is plugged in, so if the cat doesn’t knock it to the ground we will have data later.

Vince Carey (13:54:07): > oh, i got a little data – example basiliskStart fails … not clear why. must leave it, i will run by hand and try to diagnose later

Aaron Lun (14:22:50): > Good, very good.

Vince Carey (15:46:45): > I hate to say this but example(basiliskStart) destroys R 4.0.0 r78116 of 3-31 on windows 10. I will communicate with you off line.

Aaron Lun (15:48:42): > WOAH.

Aaron Lun (15:49:15): > Yes, I thought that the R process was being shut down, that would explain the lack of error messages.

Aaron Lun (15:49:48): > Good, good. We’re getting closer.

Vince Carey (15:55:10): > I sent you email at infinite.monkeys with a transcript

Vince Carey (15:55:54): > I hesitate to update my R on windows because I think I might have to reinstall all of Bioconductor but if this is necessary@Hervé Pagèslet me know.

Hervé Pagès (16:36:21): > It is not necessary, only ****strongly**** recommended. What happens if you don’t do it is unpredictable/undefined. So updating your packages after you update R is always the right thing to do. Now that the Windows binaries are available, doing it on this platform is very fast anyway (takes a few minutes only). It’s not like on Linux where it takes between 2h and 3h to reinstall everything (that’s on my laptop with a crappy internet).

Aaron Lun (16:37:50): > @Vince Careyare you available for a bit? I will try to fiddle with stuff and see if anything sticks.

Vince Carey (16:38:12): > yes, if you have some git commits i can pull them in and try again

Vince Carey (16:38:36): > just send a slack note

Aaron Lun (16:38:45): > I’ll take this to the DM’s.

Al J Abadi (19:05:22): > Hi@Hervé Pagès, > We received an email from@Nitesh Turagahours ago about a build failure of mixOmics package while our last build on the 19th was successful. I think it can be confusing to some developers.

Hervé Pagès (19:09:18): > That is not an automatic email.@Nitesh Turagais a REAL person and he’s contacting you about your package. You should discuss this with him. It has nothing to do with the automatic failure notifications which come fromBBS-noreply@bioconductor.organd start with > > [This is an automatically generated email. Please don't reply.] > > Thanks!

Al J Abadi (19:10:10): > I see, that makes sense. Thanks!

Aaron Lun (19:11:47): > if it’s any consolation, someone thought I was a GitHub bot once.

Al J Abadi (19:12:41): > Because of your commit frequency, probably?

2020-04-28

Lori Shepherd (18:51:23): > Bioconductor 3.11 is Released! Thank you to all developers and community members for contributing to the project! lease see the full release announcement:https://bioconductor.org/news/bioc_3_11_release/

Aaron Lun (18:51:56): > :+1:

2020-04-30

Michael Stadler (03:03:42): > @Michael Stadler has joined the channel

Aaron Lun (19:12:23): > YES! YES! basilisk builds with pip and Python 2.7 support on Windows!

Aaron Lun (19:12:39): > @FelixErnst@Vince Carey

Aaron Lun (19:12:49): > http://bioconductor.org/checkResults/devel/bioc-LATEST/basilisk/

Aaron Lun (19:13:10): > Insane. Now there is only oneskip_on_os("windows")test and that’s because the path in the check directory is too long.

Peter Hickey (20:06:27): > i admire your persistence, Aaron:muscle:

2020-05-01

Aaron Lun (17:26:36): > Having said that, the failure in machv2 does not look like a sporadic thing.

Aaron Lun (17:27:44): > Is this a new machine? I thought the mac builder had a fancier name.

Aaron Lun (17:59:11): > Hm. Check runs fine on my local Mac with R 4.0 from the binary installer.

Aaron Lun (17:59:31): > @Hervé Pagèsare there any interesting differences between machv1 and the old mac builder (merida?)

Hervé Pagès (18:15:47): > The name: machv2 (Mojave) is the host that was running old celaya2 builder (El Capitan) via a VM layer. After celaya2 died and we failed to resuscitate it, and after hearing from Simon Urbanek that they were finally switching from El Capitan to High Sierra (or higher) for the Mac builds, we got rid of the VM layer and we now run the builds directly on the native OS. > Interesting differences: merida1 (like merida2) was running El Capitan. machv2 is running Mojave. So almost everything is different. The compilers, the system libraries, and almost anything we install via brew or other mechanism are more recent. The compilers are the biggest difference: at the same time Simon decided to switch from El Capitan to High Sierra he also decided to drop the customized compilers they’d been using for years and to use Apple’s compilers instead.

Aaron Lun (18:16:53): > Hmmm…

Aaron Lun (18:18:44): > I think I might need some debugging assistance for basilisk on machv2, I can’t reproduce it locally on my mac with the standard R 4.0 install.

Hervé Pagès (18:24:40): > I don’t know. Like what kind of debugging assistance?

Aaron Lun (18:26:04): > Just running: > > library(basilisk) > example(basiliskStart) > > would be informative. I can see that’s the first failure point in the check report.

Aaron Lun (18:27:44): > My guess would be that it’s some kind of shared library problem that causes R to exhibit a hard crash.

Hervé Pagès (18:28:54): > > > library(basilisk) > > example(basiliskStart) > > bslskS> # Loading the base environment: > bslskS> cl <- basiliskStart(NULL) > Killed: 9 > > that was easy enough

Aaron Lun (18:29:08): > Oh man. That’s the worst case scenario.

Aaron Lun (18:30:44): > Hm. It seems it doesn’t even start trying to install Anaconda.

Hervé Pagès (18:33:03): > Can I logout? I cannot maintain this VPN thing up and running for too long because it slows down all my other accesses to internet. Thanks to the Cisco AnyConnect crap thing.

Aaron Lun (18:33:25): > oh. hold on, if you can, I’ll give you one more example.

Aaron Lun (18:33:29): > Let me just check it works on my end.

Aaron Lun (18:34:10): > > reticulate::use_condaenv(basilisk.utils::getBasiliskDir(), required=TRUE) > reticulate::py_config() >

Hervé Pagès (18:34:27): > Cisco AnyConnect is so bad that it doesn’t let me do video calls.

Aaron Lun (18:38:02): > back in my day, we didn’t have video calls.

Aaron Lun (18:38:21): > We had letters. And quills.

Hervé Pagès (18:38:32): > back in my day we didn’t have the coronavirus

Hervé Pagès (18:39:34): > So, does it work on your end?

Hervé Pagès (18:39:44): > FWIW here is what I get on machv2: > > > library(basilisk) > > reticulate::use_condaenv(basilisk.utils::getBasiliskDir(), required=TRUE) > > reticulate::py_config() > Killed: 9 >

Aaron Lun (18:39:48): > INTERESTING.

Aaron Lun (18:39:51): > Very interesting.

Aaron Lun (18:40:05): > Okay, thanks, I will spend some time pondering this.

Aaron Lun (18:40:23): > The above is basically naked reticulate operating on the conda installation; no basilisk involved.

Aaron Lun (18:40:49): > Good. It’s not my fault, then.

Hervé Pagès (18:41:08): > I guess that’s kind of good news, but also bad news cause now you need to get this fixed upstream

Hervé Pagès (18:42:08): > I’m leaving machv2 now

Aaron Lun (18:42:20): > okay, thanks.

Hervé Pagès (18:43:12): > no problem. I wish all my debugging sessions were that easy:smiley:

Aaron Lun (19:30:56): > What’s the timeframe for Mac builds on Bioc-Devel? Hopefully the error reproduces there so I don’t have to test fixes in release.

Hervé Pagès (19:38:42): > We still need a plan, we don’t have a machine yet. Might take a while…

Aaron Lun (19:39:00): > Bum.

Aaron Lun (19:39:14): > oh well. guess people can just installbasiliskfrom source on a mac for the time being.

Hervé Pagès (19:44:39): > sure but … let me see if I’m not missing something here. (1) That doesn’t help with testing fixes in devel (unless you were not referring to Mac specific fixes). (2) IIUC basilisk’s binaries are thin and all the Anaconda (miniconda? gosh, so many snake names to remember) stuff actually gets downloaded and installed on the user machine at load time right?

Hervé Pagès (19:45:18): > Or even delayed more e.g. until first usage

Aaron Lun (20:49:49): > When we can’t provide binaries, I assume the installer will ask whether the user wants to install from source. In the case of basilisk, this is possible because it’s basically just a R package, no native code involved. The user can then install basilisk on their mac and proceed to use downstream code - which at least works for me on High Sierra.

2020-05-02

Hervé Pagès (01:21:43): > Right. So from a user point-of-view and in the case of basilisk it shouldn’t make any difference whether they install the binary or the source.

Aaron Lun (01:22:35): > That’s right - provided that the user doesn’t have whatever setting is causing basilisk to fail on machv2.

Hervé Pagès (01:24:13): > But let’s say they have those settings, it will still not make any difference whether they install the binary or the source. Sorry I’m still a bit confused.

Aaron Lun (01:24:53): > Well, if they have those settings, they can install basilisk fine, but the R session will presumably just crash whenever they try to do anything with basilisk.

Hervé Pagès (01:26:10): > sure sure, I understand, and that will happen whether they installed the source or the binary

Aaron Lun (01:26:24): > Yep.

Hervé Pagès (01:27:20): > ok, thanks, I wasn’t sure I was missing something important

Vince Carey (18:28:50): > I am using mojave and > > yaml pkgs/main/osx-64::yaml-0.1.7-hc338f04_2 > zeromq pkgs/main/osx-64::zeromq-4.3.1-h0a44026_3 > zict pkgs/main/noarch::zict-1.0.0-py_0 > zipp pkgs/main/noarch::zipp-0.6.0-py_0 > zlib pkgs/main/osx-64::zlib-1.2.11-h1de35cc_3 > zstd pkgs/main/osx-64::zstd-1.3.7-h5bba6e5_0 > > > Preparing transaction: done > Executing transaction: / b'' > done > installation finished. > Killed: 9 >

Vince Carey (18:29:29): > In other words@Aaron Lunit looks like anaconda has gone well, then a crash.

Vince Carey (18:30:17): > BTW that was onbasiliskStart(NULL): > > > basiliskStart(NULL) > PREFIX=/Users/stvjc/Library/Caches/basilisk/1.1.1/0 > Unpacking payload ... > Collecting package metadata (current_repodata.json): done > Solving environment: done > > ## Package Plan ## > > environment location: /Users/stvjc/Library/Caches/basilisk/1.1.1/0 > > added / updated specs: > - _ipyw_jlab_nb_ext_conf==0.1.0=py37_0 > - alabaster==0.7.12=py37_0 > - anaconda-client==1.7.2=py37_0 >

Vince Carey (18:30:33): > It all looked so civilized …

Aaron Lun (20:13:22): > Thanks@Vince Carey. That looks like BioC-devel. Bum, I was hoping that the activation in 1.1.1 would sort it out.

Aaron Lun (20:13:43): > Could you tryreticulate::install_miniconda()and see if you canuse_condaenvthe resulting Miniconda installation?

Hervé Pagès (20:58:11): > Good. So it’s not my fault either:relieved:

2020-05-03

Vince Carey (06:25:17): > > R version 4.0.0 Patched (2020-04-27 r78309) -- "Arbor Day" > Copyright (C) 2020 The R Foundation for Statistical Computing > Platform: x86_64-apple-darwin17.0 (64-bit) > > R is free software and comes with ABSOLUTELY NO WARRANTY. > You are welcome to redistribute it under certain conditions. > Type 'license()' or 'licence()' for distribution details. > > Natural language support but running in an English locale > > R is a collaborative project with many contributors. > Type 'contributors()' for more information and > 'citation()' on how to cite R or R packages in publications. > > Type 'demo()' for some demos, 'help()' for on-line help, or > 'help.start()' for an HTML browser interface to help. > Type 'q()' to quit R. > > > library(reticulate) > 1/4 packages newly attached/loaded, see sessionInfo() for details. > > py_config() > Killed: 9 > > my suspicion is that this has to do with notarization

FelixErnst (06:41:44): > > > library(reticulate) > > py_config() > ... > * Miniconda has been successfully installed at '/Users/ernstf/Library/r-miniconda'. > python: /Users/ernstf/Library/r-miniconda/envs/r-reticulate/bin/python > libpython: /Users/ernstf/Library/r-miniconda/envs/r-reticulate/lib/libpython3.6m.dylib > pythonhome: /Users/ernstf/Library/r-miniconda/envs/r-reticulate:/Users/ernstf/Library/r-miniconda/envs/r-reticulate > version: 3.6.10 |Anaconda, Inc.| (default, Mar 25 2020, 18:53:43) [GCC 4.2.1 Compatible Clang 4.0.1 (tags/RELEASE_401/final)] > numpy: /Users/ernstf/Library/r-miniconda/envs/r-reticulate/lib/python3.6/site-packages/numpy > numpy_version: 1.18.1 >

FelixErnst (06:41:48): > Works for me

FelixErnst (06:41:57) (in thread): > > > sessionInfo() > R version 4.0.0 (2020-04-24) > Platform: x86_64-apple-darwin17.0 (64-bit) > Running under: macOS Catalina 10.15.4 > > Matrix products: default > BLAS:  /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib > LAPACK: /Library/Frameworks/R.framework/Versions/4.0/Resources/lib/libRlapack.dylib > > locale: > [1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8 > > attached base packages: > [1] stats    graphics grDevices utils    datasets methods  base     > > other attached packages: > [1] reticulate_1.15 > > loaded via a namespace (and not attached): > [1] compiler_4.0.0 Matrix_1.2-18  tools_4.0.0    rappdirs_0.3.1 Rcpp_1.0.4.6   grid_4.0.0     > [7] jsonlite_1.6.1 lattice_0.20-41 >

FelixErnst (06:42:22): > but I didn’t use R-devel

FelixErnst (06:46:24): > I guess on machv2 R-release is used as well. However, the error of basilisk is reproducable locally on my macOS installation (I send@Aaron Lunthe output)

FelixErnst (06:48:19): > … and the error message differs a bit: > > bslskS> cl <- basiliskStart(tmploc) > Error in makePSOCKcluster(1) : > Cluster setup failed. 1 worker of 1 failed to connect. >

Vince Carey (08:40:38): > I don’t think R-devel should be used at this point.

Vince Carey (08:42:34): > With the version of R distributed athttp://mac.r-project.org/high-sierra/R-4.0-branch/x86_64/R-4.0-branch.tar.gzI make more progress – it isn’t immediately killed. > > %vjcair> ./R --vanilla > > R version 4.0.0 Patched (2020-04-27 r78309) -- "Arbor Day" > Copyright (C) 2020 The R Foundation for Statistical Computing > Platform: x86_64-apple-darwin17.0 (64-bit) > > R is free software and comes with ABSOLUTELY NO WARRANTY. > You are welcome to redistribute it under certain conditions. > Type 'license()' or 'licence()' for distribution details. > > Natural language support but running in an English locale > > R is a collaborative project with many contributors. > Type 'contributors()' for more information and > 'citation()' on how to cite R or R packages in publications. > > Type 'demo()' for some demos, 'help()' for on-line help, or > 'help.start()' for an HTML browser interface to help. > Type 'q()' to quit R. > > > library(basilisk) > > basiliskStart(NULL) > WARNING: ignoring environment value of R_HOME > Error in unserialize(node$con) : error reading from connection > In addition: Warning message: > Python '/Users/stvjc/Library/Caches/basilisk/1.0.0/0/bin/python' was requested but '/Users/stvjc/Library/r-miniconda/envs/r-reticulate/bin/python' was loaded instead (see reticulate::py_config() for more information) >

FelixErnst (10:36:49): > R version 4.0.0 Patched (2020-04-27 r78309) -- "Arbor Day"is R-devel isn’t it?#

FelixErnst (10:37:17): > R version 4.0.0 (2020-04-24)Is release

FelixErnst (10:37:34): > or am I missing something?

Vince Carey (10:55:45): > no, R-devel is 4.1.0 at this time.

FelixErnst (11:00:06): > Sorry but I see a revision number. That means that it is devel, isn’t it? R trunk probably will become R 4.1 in year, but more likely it will become 4.0.1 in a few montah > A R 4.1 branch does not existhttps://svn.r-project.org/R/trunk/https://svn.r-project.org/R/branches/R-4-0-branch/they are the same arent’t they?

Vince Carey (11:03:06): > > %rex2> svn info > Path: . > URL:[https://svn.r-project.org/R/trunk](https://svn.r-project.org/R/trunk)Repository Root:[https://svn.r-project.org/R](https://svn.r-project.org/R)Repository UUID: 00db46b3-68df-0310-9c12-caf00c1e9a41 > Revision: 77425 > Node Kind: directory > Schedule: normal > Last Changed Author: luke > Last Changed Rev: 77425 > Last Changed Date: 2019-11-15 14:14:54 -0500 (Fri, 15 Nov 2019) > > %rex2> cat VERSION > 4.1.0 Under development (unstable) >

FelixErnst (11:03:13): > hmm

Vince Carey (11:06:09): > this may be more convincing as i svn-upped the whole folder > > %rex2> svn info > Path: . > URL:[https://svn.r-project.org/R/trunk](https://svn.r-project.org/R/trunk)Repository Root:[https://svn.r-project.org/R](https://svn.r-project.org/R)Repository UUID: 00db46b3-68df-0310-9c12-caf00c1e9a41 > Revision: 78347 > Node Kind: directory > Schedule: normal > Last Changed Author: hornik > Last Changed Rev: 78347 > Last Changed Date: 2020-05-03 07:49:34 -0400 (Sun, 03 May 2020) > > %rex2> cat VERSION > 4.1.0 Under development (unstable) >

Vince Carey (11:06:49): > R-release will proceed with 4.0.1, etc., and in 6 months or so we will transition R 4.1.x to release

FelixErnst (11:07:34): > So you build r78309 from source I guess

Vince Carey (11:09:19): > No I have not tried that yet. I used this, tar zxf after wgethttp://mac.r-project.org/high-sierra/R-4.0-branch/x86_64/R-4.0-branch.tar.gzand then reset R_HOME in the R script. That seems unsound owing to the warning I got concerning ignoring environment variable R_HOME… But I do not want to have to reinstall everything.

FelixErnst (11:11:33): > Now I get it. Thanks for the clarification

FelixErnst (11:13:32): > I wanted to use the R versioned used on the machv2 environment and so I was looking for the date as well as the version number4.0.0 (2020-04-24)and your4.0.0 Patched (2020-04-27 r78309)tripped me up

Martin Morgan (11:53:33) (in thread): > There will be minor version release of R-4.0, e.g., R-4.0.1, R-4.0.2, … These will be from commits to R-4-0-branch (i.e., bug fixes and minor changes) and not from R-devel (new features). Commits to R-devel will only be ‘released’ as R-4.1.0 in about a year.

Hervé Pagès (15:10:19): > For more meaningful testing and comparison it’s better to stick to R 4.0.0 and use the official Mac binary from CRAN (https://cran.r-project.org/bin/macosx/).

Vince Carey (16:57:09): > @Hervé Pagèsare you able to reproduce > > > sessionInfo() > R version 4.0.0 Patched (2020-04-27 r78309) > Platform: x86_64-apple-darwin17.0 (64-bit) > Running under: macOS Mojave 10.14.6 > > Matrix products: default > BLAS: /Library/Frameworks/R.framework/Versions/4.0/Resources/lib/libRblas.dylib > LAPACK: /Library/Frameworks/R.framework/Versions/4.0/Resources/lib/libRlapack.dylib > > locale: > [1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8 > > attached base packages: > [1] stats graphics grDevices utils datasets methods base > > other attached packages: > [1] reticulate_1.15 rmarkdown_2.1 > > loaded via a namespace (and not attached): > [1] compiler_4.0.0 startup_0.14.1 Matrix_1.2-18 tools_4.0.0 > [5] htmltools_0.4.0 Rcpp_1.0.4.6 codetools_0.2-16 grid_4.0.0 > [9] knitr_1.28 jsonlite_1.6.1 xfun_0.13 digest_0.6.25 > [13] rlang_0.4.5 lattice_0.20-41 evaluate_0.14 > > py_config() > Killed: 9 >

Hervé Pagès (16:57:45): > Starting VPN now and going on machv2…

Vince Carey (16:58:29): > I can confirm that on a freshly installed catalina and the unnotarized binary R fromhttp://mac.r-project.org/high-sierra/R-4.0-branch/x86_64/R-4.0-branch.tar.gzboth basilisk and BiocSklearn run fine. I think this is the walled garden problem. This may be why Simon changed to the Apple toolchain.

Vince Carey (17:01:03): > The notarization is mentioned herehttps://sandbox.anaconda.com/blog/how-to-restore-anaconda-after-macos-catalina-update… as long as we use the right python infrastructure we will be OK? - Attachment (Anaconda): How to Restore Anaconda after Update to MacOS Catalina | Anaconda > MacOS Catalina was released on October 7, 2019, and has been causing quite a stir for Anaconda users. Apple has decided that Anaconda’s default install location in the root folder is not allowed. It moves that folder into a folder on your desktop called “Relocated Items,” in the Security folder. If…

Aaron Lun (17:01:17): > Just catching up to this.

Aaron Lun (17:02:20): > We shouldn’t be putting stuff in the root directory anyway, it goes into the appdirs cache.

Aaron Lun (17:02:56) (in thread): > I mean, this looks particularly bad.

Hervé Pagès (17:05:40) (in thread): > Here is what I get on machv2: > > > library(reticulate) > > py_config() > python: /usr/local/bin/python3 > libpython: /usr/local/opt/python/Frameworks/Python.framework/Versions/3.7/lib/python3.7/config-3.7m-darwin/libpython3.7.dylib > pythonhome: /usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7:/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7 > version: 3.7.7 (default, Mar 10 2020, 15:43:03) [Clang 11.0.0 (clang-1100.0.33.17)] > numpy: /usr/local/lib/python3.7/site-packages/numpy > numpy_version: 1.18.2 > > python versions found: > /usr/local/bin/python3 > /usr/bin/python > > sessionInfo: > > > sessionInfo() > R version 4.0.0 (2020-04-24) > Platform: x86_64-apple-darwin17.0 (64-bit) > Running under: macOS Mojave 10.14.6 > > Matrix products: default > BLAS: /Library/Frameworks/R.framework/Versions/4.0/Resources/lib/libRblas.dylib > LAPACK: /Library/Frameworks/R.framework/Versions/4.0/Resources/lib/libRlapack.dylib > > locale: > [1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8 > > attached base packages: > [1] stats graphics grDevices utils datasets methods base > > other attached packages: > [1] reticulate_1.15 > > loaded via a namespace (and not attached): > [1] compiler_4.0.0 Matrix_1.2-18 rappdirs_0.3.1 Rcpp_1.0.4.6 > [5] grid_4.0.0 jsonlite_1.6.1 lattice_0.20-41 >

Hervé Pagès (17:08:24): > @Vince Careysee my answer in thehttps://community-bioc.slack.com/archives/CEQ04GKEC/p1588539776347100?thread_ts=1588501517.322800&cid=CEQ04GKECthread - Attachment: Attachment > I mean, this looks particularly bad.

Aaron Lun (17:16:45): > So… this seems like a reticulate/conda problem, then.

Hervé Pagès (17:24:50): > Let’s keep this conversation in the thread.

Aaron Lun (17:25:05): > I was talking more generally, but sure.

Hervé Pagès (17:26:40) (in thread): > > So… this seems like a reticulate/conda problem, then. > I have no idea but FWIW I did NOT get a Killed: 9

Aaron Lun (17:27:04) (in thread): > Right, but that’s because you were pulling a non-conda python.

Aaron Lun (17:27:48) (in thread): > Did anyone try the pure-reticulate approach withinstall_miniconda()? I see it went fine for@FelixErnst.

Hervé Pagès (17:29:28) (in thread): > Just ran the code that@Vince Careyasked me to run but if there is something else I should have run please tell me what it is.

Hervé Pagès (17:30:04) (in thread): > ha so that’s itinstall_miniconda()

Aaron Lun (17:30:55) (in thread): > Perhaps try: > > install_miniconda() > use_condaenv("wherever that miniconda ended up") > py_config() >

Hervé Pagès (17:32:41) (in thread): > It’s Sunday and my brain is on weekend so I’m just copy/past’ing whatever you guys put here while keeping my eyes closed at the same time (I don’t want to read it).

Hervé Pagès (17:33:51) (in thread): > I should be able to ask my daughter to do the copy/paste for me

Hervé Pagès (17:34:07) (in thread): > and copy/paste the output back in Slack

Aaron Lun (17:34:48) (in thread): > Uh, hold on, then. The path should be something like/Users/ernstf/Library/r-miniconda/envs/r-reticulate/

Hervé Pagès (17:41:38) (in thread): > install_miniconda()returns this path so there should be a way to programmatically pass it touse_condaenv():wink:Here is goes on machv2: > > > library(reticulate) > > install_miniconda() > ... > [lots of things going on] > ... > Executing transaction: done > # > # To activate this environment, use > # > # $ conda activate r-reticulate > # > # To deactivate an active environment, use > # > # $ conda deactivate > > * Miniconda has been successfully installed at '/Users/biocbuild/Library/r-miniconda'. > [1] "/Users/biocbuild/Library/r-miniconda" > > use_condaenv("/Users/biocbuild/Library/r-miniconda") > > py_config() > Killed: 9 > > So yes, another “Killed: 9”

Aaron Lun (17:44:54) (in thread): > oh thank god.

Aaron Lun (17:45:04) (in thread): > Itisreticulate’s fault, then.

Aaron Lun (17:45:12) (in thread): > Or at least conda’s fault.

Hervé Pagès (17:45:56) (in thread): > Looks like it. Looks like we’ve completely left basilisk out of the picture here…

Aaron Lun (17:46:28) (in thread): > Currently searching reticulate’s GitHub to see if they have open issues about this.

Aaron Lun (17:48:01) (in thread): > Looks likehttps://github.com/rstudio/reticulate/issues/758. Perhaps we should chip in with our experiences on the build system.

Hervé Pagès (17:48:57) (in thread): > Last time I checked packages in the reticulate stack like tensorflow they were skipping almost all tests and examples so they would pass CHECK on the CRAN build machines. So no surprise everything looks good on CRAN’s side.

Aaron Lun (17:49:44) (in thread): > LOL.

Vince Carey (17:50:53) (in thread): > What about finding a notarized miniconda from anaconda?

Aaron Lun (17:52:21) (in thread): > I don’t know what that means. There’s only one set of ana/miniconda installers AFAIK.

Vince Carey (18:04:38) (in thread): > I added two comments at issue 758 of reticulate, indicating where the non-notarized R can be found, and the issues with miniconda.

Aaron Lun (18:14:38) (in thread): > Thanks vince

2020-05-05

Hervé Pagès (14:30:55): > <!channel>The release builds (3.11 builds) are experiencing problems since last Friday (April 30) which is why the latest published report is still from that day:https://bioconductor.org/checkResults/3.11/bioc-LATEST/We’re working on it and hopefully we’ll get a new report before the end of the week. > > Thanks for your comprehension and sorry for the inconvenience.

2020-05-06

Aaron Lun (03:06:48) (in thread): > @Vince CareyIf it is a MKL problem, I wonder whether: > > path <- reticulate::install_miniconda() > reticulate::conda_install(envname=path, packages=c("nomkl", "numpy")) > reticulate::use_condaenv(path, required=TRUE) > reticulate::py_config() > > would help.

Vince Carey (11:24:40) (in thread): > yes > > > reticulate::py_config() > python: /Users/stvjc/Library/r-miniconda/bin/python > libpython: /Users/stvjc/Library/r-miniconda/lib/libpython3.7m.dylib > pythonhome: /Users/stvjc/Library/r-miniconda:/Users/stvjc/Library/r-miniconda > version: 3.7.6 (default, Jan 8 2020, 13:42:34) [Clang 4.0.1 (tags/RELEASE_401/final)] > numpy: /Users/stvjc/Library/r-miniconda/lib/python3.7/site-packages/numpy > numpy_version: 1.18.4 > > NOTE: Python version was forced by use_python function >

Aaron Lun (11:44:52) (in thread): > hoho. So maybe for the time being we can get around it by forcing the issue with a no-MKL installation.

Hervé Pagès (11:53:57) (in thread): > Sounds promising. Should I test this on machv2?

Aaron Lun (11:54:50) (in thread): > Yes please. If it works I will modify basilisk to do this.

Hervé Pagès (12:26:52) (in thread): > > > path <- reticulate::install_miniconda() > Error: Miniconda is already installed at '/Users/biocbuild/Library/r-miniconda' > > How do I uninstall this miniconda?

Hervé Pagès (12:27:02) (in thread): > Just delete that folder?

Aaron Lun (12:27:06) (in thread): > yeah

Hervé Pagès (12:32:58) (in thread): > Looks good: > > > path <- reticulate::install_miniconda() > ... > ... > > reticulate::conda_install(envname=path, packages=c("nomkl", "numpy")) > ... > ... > > reticulate::use_condaenv(path, required=TRUE) > > reticulate::py_config() > python: /Users/biocbuild/Library/r-miniconda/bin/python > libpython: /Users/biocbuild/Library/r-miniconda/lib/libpython3.7m.dylib > pythonhome: /Users/biocbuild/Library/r-miniconda:/Users/biocbuild/Library/r-miniconda > version: 3.7.7 (default, Mar 23 2020, 17:31:31) [Clang 4.0.1 (tags/RELEASE_401/final)] > numpy: /Users/biocbuild/Library/r-miniconda/lib/python3.7/site-packages/numpy > numpy_version: 1.18.4 > > NOTE: Python version was forced by use_python function >

Aaron Lun (12:33:15) (in thread): > Excellent.

Aaron Lun (12:37:06) (in thread): > @Vince Careyone additional complication is that it literally takes ages to strip MKL from an Anaconda installation; I think it’s been running for 10 minutes now. This is motivating me to switch to Miniconda (tagging@FelixErnst). Currently you are the only dependency, and I think you were relying on the base scikit-learn; I think we’ll trial some stuff in devel to see if it works for you.

Vince Carey (12:40:58) (in thread): > Sure

FelixErnst (14:10:13) (in thread): > So to recap: This a problem of MKL libraries not being signed on macOS?

Aaron Lun (14:10:59) (in thread): > So it seems.

FelixErnst (14:12:13) (in thread): > and since miniconda comes without MKL, it works, but the full Anaconda includes the MKL libraries and therefore gets killed?

FelixErnst (14:12:44) (in thread): > ok, so that is a really weird one and probably a first on an R dependency tree.

Aaron Lun (14:13:50) (in thread): > Well, in the case of both the build machine and Vince’s machine, neither miniconda or anaconda worked. While it’s true that miniconda doesn’t come with MKL, when reticulate installs it viainstall_miniconda, it also installs numpy, and that pulls down MKL by default. No way to intercept that process with the currentinstall_minicondafunction, so you get it whether you like it or not.

Aaron Lun (14:28:09) (in thread): > @Vince Carey. It is done: LTLA/basilisk.utils@nomkl and LTLA/basilisk@mini are the two branches that I would like you to test.

Aaron Lun (14:29:16) (in thread): > This will require you to have Biocsklearn create its own dependencies again… let’s see how that goes.

Vince Carey (20:37:28) (in thread): > Well, skPCA works in the new regime. But > > > dem = skKMeans(iris[,1:4], n_clusters=3L) > Error in py_call_impl(callable, dots$args, dots$keywords) : > TypeError: float() argument must be a string or a number, not 'dict' > > Detailed traceback: > File "/Users/stvjc/Library/Caches/basilisk/1.1.2/BiocSklearn-1.11.1/bsklenv/lib/python3.7/site-packages/sklearn/cluster/_kmeans.py", line 859, in fit > order=order, copy=self.copy_x) > File "/Users/stvjc/Library/Caches/basilisk/1.1.2/BiocSklearn-1.11.1/bsklenv/lib/python3.7/site-packages/sklearn/utils/validation.py", line 531, in check_array > array = np.asarray(array, order=order, dtype=dtype) > File "/Users/stvjc/Library/Caches/basilisk/1.1.2/BiocSklearn-1.11.1/bsklenv/lib/python3.7/site-packages/numpy/core/_asarray.py", line 85, in asarray > return array(a, dtype, copy=False, order=order) >

Vince Carey (20:39:12) (in thread): > maybe it is pandas, i did not include that

Aaron Lun (20:41:26) (in thread): > yeah, miniconda is coming in totally fresh.

Vince Carey (20:41:41) (in thread): > Of note, I used > > /Users/stvjc/Library/Caches/basilisk/1.1.2/BiocSklearn-1.11.1/bsklenv/bin/pip install scikit-learn==0.22.2post1 > > because the installation process invoked by basilisk could not find the post1 version – and that is the one that has the sklearn alias … 0.22.2 can’t find it

Aaron Lun (20:42:45) (in thread): > I’m not sure I understand. What is this alias?

Vince Carey (20:46:33) (in thread): > I don’t understand either. You can’t say import scikit-learn. You have to say import sklearn. With reticulate > in any event, with 0.22.2, import(“sklearn”) fails. With 0.22.2post1, it succeeds.

Vince Carey (20:47:18) (in thread): > Adding pandas to my dependencies solved the problem. So this nomkl is a big win. But what will the next pitfall be?

Aaron Lun (20:48:15) (in thread): > Indeed. Interesting to see that 0.22.2 lacksimport sklearn. How else is someone meant to import it?

Aaron Lun (20:57:10) (in thread): > In any case - good - let’s get this on BioC-devel and see whether Windows will eat it.

Vince Carey (22:42:32) (in thread): > ok, i pushed

Aaron Lun (22:51:09) (in thread): > Waiting to pass CHECK locally, and then I will push as well.

2020-05-07

Aaron Lun (00:25:34) (in thread): > Clean and green on Unix and Mac.

Aaron Lun (00:25:45) (in thread): > 1.1.1 for utils, 1.1.2 for basilisk.

Steffen Neumann (03:29:50): > Hi, running the docker containerbioconductor_docker:develI am getting R4.1 and BiocManager complains. Is that known ? My error ? correct Slack channel ?

Steffen Neumann (03:49:16): > Ah, Nitesh is already on it:https://github.com/rocker-org/rocker-versioned/issues/208

Jianhong (14:18:01): > @Jianhong has joined the channel

Hervé Pagès (20:16:04) (in thread): > mmh so some packages (netReg, phemd, scBFA, schex, scMAGeCK scTensor, Spaniel) started to kill R at load time on machv2 e.g.: > > > library(netReg) > Loading required package: tensorflow > Loading required package: tfprobability > Killed: 9 > > We see this on today’s report e.g. herehttps://bioconductor.org/checkResults/3.11/bioc-LATEST/netReg/machv2-install.htmlDon’t know exactly when this started but this is very recent (we didn’t see this last week). All these packages rely directly or indirectly on reticulate. Could this be related with basilisk’s business with reticulate? Or could this be the consequence of me running the code you asked me to run recently to help with the debugging and not cleaning after that? E.g. should I remove~biocbuild/Library/r-miniconda? Should I just nuke this manually or should I use some reticulate command for this? Anything else I should do to restore the previous state? Thanks! > PS: Note that not all packages that depend directly or indirectly on reticulate have this problem though e.g. idr2d and MOFA.

Aaron Lun (21:07:35) (in thread): > Oh, yes.reticulatehas thisnastyhabit of defaulting to the miniconda that it just installed, even after you EXPLICITLY tell it to use something else. Destroy the miniconda installation directory and all should be well. Though those affected packages should really use a more robust system of getting the right python, there’s no reason why this wouldn’t happen on the user machine.

Hervé Pagès (23:44:25) (in thread): > > Destroy the miniconda installation directory and all should be well. > Done. > > Though those affected packages should really use a more robust system of getting the right python, there’s no reason why this wouldn’t happen on the user machine. > Can we point them to some concrete recommendations for this? Thanks!

Aaron Lun (23:46:07) (in thread): > WEll… basilisk would be the recommendation! Waiting to see whether the noMKL’d miniconda goes through the tokay gauntlet on BIoC-devel.

Hervé Pagès (23:59:21) (in thread): > > basilisk would be the recommendation > CRAN packages cannot be basilisk clients. Are you saying that reticulate-based CRAN packages are going to crash R at load time after people install basilisk?

2020-05-08

Aaron Lun (00:16:14) (in thread): > I thought you were talking about Bioconductor packages.

Aaron Lun (00:16:45) (in thread): > In any case, basilisk will not interfere with CRAN packages’ fragile use of reticulate.

Aaron Lun (00:17:54) (in thread): > Unless, of course, one package loads basilisk to use Python and another package does not, in which case the package that doesn’t use basilisk will simply fail because native reticulate doesn’t support use of two versions of Python in the same R session. Getting around this limitation is the whole point of basilisk.

Hervé Pagès (00:30:23) (in thread): > I’ve no experience with reticulate or basilisk so I don’t really understand the story here. I was just wondering how we could translate “packages should use a more robust system of getting the right python” into some concrete recommendation. If the only recommendation is that they should use basilisk then, fine, but that also kind of implies that there is no robust way for a CRAN package to get “the right Python”. Which is kind of worrying but maybe I should relax and just trust that everything is going to be alright.

Aaron Lun (00:32:00) (in thread): > Yes, that’s right. That’s the problem that basilisk aims to solve, so there’s nothing we can do about CRAN packages that use reticulate, I’m afraid.

Hervé Pagès (01:01:11) (in thread): > What will happen if I already loaded a reticulate-based CRAN package that uses the python that is on the PATH and then later in the same session I load a basilisk client?

Aaron Lun (01:02:13) (in thread): > Loading the packages into the session won’t do anything, provided no one is doing naughty things in their.onLoad.

Aaron Lun (01:03:06) (in thread): > I assume you’re talking about actually using those packages to run stuff, in which case the order you’ve described above will be fine; the CRAN package will populate the single Python slot for this current session with the system Python, and the basilisk-dependent package will detect that and fork appropriately.

Hervé Pagès (01:03:30) (in thread): > Well, I mean I’ve already loaded a reticulate-based CRAN and doing things with it so reticulate is already locked with a given python

Hervé Pagès (01:03:43) (in thread): > yes

Hervé Pagès (01:05:57) (in thread): > but basilisk clients need modules that are not necessarily installed for that python

Hervé Pagès (01:09:00) (in thread): > This also means that the python I get depends in the order in which I load (and use) packages doesn’t it?

Aaron Lun (01:09:33) (in thread): > are you asking with or without basilisk?

Hervé Pagès (01:10:33) (in thread): > I’m asking with 2 reticulate-based packages, one from CRAN and one basilisk client

Aaron Lun (01:15:26) (in thread): > basilisk clients always have access to all of the Python packages that they need, because basilisk creates isolated conda environments for those clients. So the basilisk clients will always work, no matter what other Python-related activity is going on in that session. > > Now, basilisk may or may not have the opportunity to occupy the shared Python slot. If it’s free, then basilisk will use it by default, which will break other reticulate-dependent packages that are not basilisk clients. However, I don’t think this is any fault of basilisk; this would have been a problemanyway for anyone wanting to use multiple environments in the same R session (and the environments are the recommended way to use reticulate). > > Nonetheless, if one wanted basilisk to be “polite”, one could setsetBasiliskShared(FALSE)to force it to never use the shared python slot.

Hervé Pagès (01:18:15) (in thread): > what happens to basilisk clients in the case that the python slot is already occupied by the non-basilisk python e.g. the system wide python?

Aaron Lun (01:20:55) (in thread): > No problem.basiliskdetects this and starts a separate R process in which it performs all the requested operations, returning the results to the parent. This is obviously less efficient sobasiliskwill always try to take the slot if it’s available, but at least it keeps on running even when it’s lost.

Hervé Pagès (01:22:26) (in thread): > Ah ok, I guess that what you meant earlier with “the basilisk-dependent package will detect that and fork appropriately”, got it now. Thanks!

Aaron Lun (21:33:52) (in thread): > Well. Fine on linux as usual, choked on tokay2 with a TIMEOUT.

Aaron Lun (21:35:54) (in thread): > Looks like the time-out is happening inbasiliskStart. Guess that makes sense.

Aaron Lun (21:48:45) (in thread): > Not quite sure why miniconda takes longer than anaconda to install here. I can only guess it’s spending more time trying to solve the environment for Windows.

Aaron Lun (21:49:26) (in thread): > Or there are connection issues from tokay2.

Aaron Lun (21:55:47) (in thread): > @Vince Careylooks like we’ll have to trade in the fix on Mac for a failure on Windows.

Vince Carey (21:57:30) (in thread): > do i need to get out the box again

Vince Carey (21:57:48) (in thread): > R or conda problem?

Aaron Lun (21:58:46) (in thread): > I don’t know, the time-out occurs inbasiliskStartwhich lies at the intersection of the two.

Aaron Lun (21:59:41) (in thread): > It could be: (i) connectivity issues for the initial Miniconda, (ii) connectivity issues for the subsequent environment creation, or (iii) a stalled socket process.

Aaron Lun (22:00:17) (in thread): > I am able to check it fine on my Windows 10 VM (latex errors notwithstanding), and I imagine you would too. I’ll try again tonight.

2020-05-09

Aaron Lun (00:02:14) (in thread): > IT IS DONE. Let’s test in RELEASE_3_11!

Aaron Lun (00:02:56) (in thread): > Hopefully Mac works okay with the no-MKL. Windows… I dunno.

Aaron Lun (00:54:18) (in thread): > OH DAMN! I realized what the problem is. IT’S WAITING FOR A PROMPT!

Aaron Lun (00:54:23) (in thread): > HAHAHA.

Aaron Lun (04:11:13) (in thread): > Well. I added some extra flags to the miniconda installer to avoid trying to add itself to the registry. I don’t know if that’ll fix the problem, because sometimes the waiting happens and sometimes it doesn’t - I could never nail down if it was indeed waiting for something - but let’s see.

Aaron Lun (04:12:35) (in thread): > Hm. Though come to think of it, what I thought was CHECK waiting for a prompt may just have been Window’s CLI refusing to print stuff in a timely manner.

2020-05-10

Sangram Keshari Sahu (09:30:52): > @Sangram Keshari Sahu has joined the channel

2020-05-11

Aaron Lun (00:22:06) (in thread): > Huzzah! Builds are back on Machv2!

Aaron Lun (00:22:20) (in thread): > Though we lost Windows builds… honestly. This is just crazy.

2020-05-14

Hervé Pagès (21:41:39): > <!channel>We finally have a Mac builder for the BioC 3.12 builds. This is just good ol’ merida1 with a fresh Mojave install and a fresh reinstall of the build system environment. If everything goes as expected it should be included in tomorrow’s report for 3.12 here:https://bioconductor.org/checkResults/3.12/bioc-LATEST/A few components that are required by a small number of BioC packages are still missing on the machine and will be installed in the next few weeks.

2020-05-17

Aaron Lun (03:20:13): > basiliskis failing onmerida1despite succeeding onmachv2. Initial suspicion is that the former is not granting permissions to editrappdirs::user_cache_dir, in this case,/Users/biocbuild/Library/Caches/basilisk/1.1.5.

Aaron Lun (03:21:30): > basilisk.utils1.1.5 should provide some insight, I have inserted an explicit error message when the directory is not created.

Hervé Pagès (20:31:46): > oops.. for some reason/User/biocbuild/Library/Cacheswas belonging to root on merida1: > > merida1:~ biocbuild$ whoami > biocbuild > merida1:~ biocbuild$ ls -al Library/Caches/ > total 0 > drwxr-xr-x 3 root staff 96 May 13 20:34 . > drwx------ 25 biocbuild staff 800 May 14 21:21 .. > drwxrwxr-x 21 biocbuild staff 672 May 14 14:58 Homebrew > > Not sure how I ended up with something like that. Just changed it. Looks like this was affecting other packages e.g. biomaRt.

2020-05-18

Aaron Lun (01:49:21): > okay, great.

Huipeng Li (11:04:04): > @Huipeng Li has joined the channel

Aaron Lun (16:36:01): > @Vince Carey@Hervé Pagès@FelixErnstI’ve done it:http://bioconductor.org/checkResults/devel/bioc-LATEST/basilisk/

Aaron Lun (16:36:04): > OMG it’s beautiful

Aaron Lun (16:36:12): > And also hilarious:

Aaron Lun (16:36:36): > * On Linux it runs the latest miniconda. > * On Mac it runs the latest miniconda without MKL. > * On Windows it runs the second-latest Anaconda.

Aaron Lun (16:37:06): > This is the working combination that I eventually figured out by trial and error.

Aaron Lun (16:37:51): > To be clear, the trial and error is only on the BBS. Everyone worked fine on my local machines and VMs.

Vince Carey (16:37:57): > Wow. BiocSklearn is green too. Hats off!

Hervé Pagès (17:32:48): > Yes it looks beautiful. I’m also a big fan of green LEDs on the build report. > > Everyone worked fine on my local machines and VMs. > Even on your local Windows?:grin:

Aaron Lun (17:33:13): > well, the VM running windows on my laptop, yes.

Hervé Pagès (18:14:36): > that is great news, even better than basilisk being all green today:wink:

Aaron Lun (22:12:45): > Took a screenshot for posterity. We may never see it again. - File (PNG): Screen Shot 2020-05-18 at 7.12.12 PM.png

Hervé Pagès (23:29:42): > my recently revised color scheme for the 3.11 builds looks really good on that screenshot:sunglasses:

2020-05-20

Hervé Pagès (23:22:10): > <!channel>As you’ve probably noticed already we didn’t have a new build report for the 3.11 software builds since Monday (May 18). We’re working on it and hopefully this will get resolved before the end of the week. Thanks for your patience.

2020-06-06

Olagunju Abdulrahman (19:56:55): > @Olagunju Abdulrahman has joined the channel

2020-06-11

Kozo Nishida (03:13:32): > @Kozo Nishida has joined the channel

2020-06-12

Aaron Lun (14:14:08): > How frequently do the dependency badges get updated?

Lori Shepherd (14:30:04) (in thread): > once a month on the first of the month. We assumed the dependencies wouldn’t be in flux that much

Aaron Lun (14:31:58) (in thread): > okay. Looks I’ll have to wait for my sweet dependency shrinkage.

2020-06-27

Hervé Pagès (18:26:36): > With joda and JunctionSeq now being deprecated we’re going to loose the only 2 packages in the J section (https://bioconductor.org/checkResults/3.12/bioc-LATEST/#J). Anyone’s considering filling the gap with a new submission?

2020-07-02

Aaron Lun (17:22:40) (in thread): > Still not updating. I should be seeing 45 packages for scran: > > > library(BiocPkgTools) > > dep_df = buildPkgDependencyDataFrame(c("Depends", "Imports")) > > > g = buildPkgDependencyIgraph(dep_df) > > length(subcomponent(g, "scran", mode="out")) > [1] 45 >

Aaron Lun (17:22:53) (in thread): > on Bioc-devel, at least.

Lori Shepherd (18:16:14) (in thread): > I’ll check tomorrow

2020-07-04

Umar Ahmad (08:22:19): > @Umar Ahmad has joined the channel

Mike Jiang (19:56:28): > cytolib is failing on merida1 (mac build server) due to the lacking some c++ filesystem library support (require c++17) > > error: 'path' is unavailable: introduced in macOS 10.15 > auto h5path = fs::path(h5_filename); > > Is it possible to build xcode with c++17 enabled?or even update maos to 10.15?

2020-07-06

Lori Shepherd (10:31:58) (in thread): > I’ll look into it more. Bioconductor does not use BiocPkgTools to build its dependency count. We utilize the tools::package_dependencies function. When I run it manually it seems like it should be 56 for scran in devel but for some reason when the views are currently generated it is showing 102 — I’ll try and investigate where the bug is.

Aaron Lun (11:18:54) (in thread): > thanks

Hervé Pagès (13:15:28): > @Mike JiangI don’t know how to build Xcode with C++17 enabled. Note that we follow CRAN’s lead for what we support on macOS. This means that we must support High Sierra (or higher) and use Apple’s standard compilers. So package code is expected to compile out of the box on that setup. Otherwise, we’re going to make things very difficult for the end users.

Mike Jiang (13:32:55) (in thread): > Ok. Fair enough. I will roll back to use the boost::filesystem for now until these c++17 features are fully supported on default mac setup.

2020-07-08

FelixErnst (00:26:58): > Hi everyone, I mentioned the the error message from a Travis-CI build on the bioc-devel list regarding the dependency issueAnnotationHubData->rBiopaxParser->nemwithnembeing deprecated for Bioc 3.12. The same error should also appear on the bioc build server, but it doesn’t. However, using the bioc docker image for devel I cannot install RNAmodR.Data (and all the reverse dependencies of it). Is there anything I can do? I mean I cannot removeAnnotationHubDatafrom the dependency of RNAmodR.Data, can I?

Mike Smith (03:58:13) (in thread): > I guess you can force an installation ofrBiopaxParserfrom the Bioc git repo, sincenemis just a suggestion. I presume that’s why the issue doesn’t show up on the build system, since the packages are cloned and built rather than viaBiocManager::intall()(I think that’s correct anyway).

Mike Smith (04:00:53) (in thread): > That’s not really a feasible long term solution, but it looks like there’s already an issue on Github (https://github.com/frankkramer-lab/rBiopaxParser/issues/6), so I’m not sure there’s much more you can do to prod the rBioxParser maintainer into action - although personally the more comments on an issue I get, the more likely I am to react to it!

Lori Shepherd (08:04:38) (in thread): > I’ll investigate the AnnotationHubData dependency and why that would be the case too.

FelixErnst (11:52:25) (in thread): > Thanks for the reply. For me the immediate issue was more on the “this shouldn’t build on bioc” (meaning RNAmodR.Data). But the next things is of course getting to the solution part of the problem. I will head over to github now…

Hervé Pagès (21:36:46) (in thread): > As Mike explained, RNAmodR and AnnotationHubData do not fail on the build machines because these machines install other Bioconductor software packages directly fromgit.bioconductor.org. So rBiopaxParser actually does get installed there.

2020-07-10

Aaron Lun (04:16:00) (in thread): > So. Any progress?

Genevieve Stein-O’Brien (15:57:49): > @Genevieve Stein-O’Brien has joined the channel

2020-07-13

Aaron Lun (20:01:05): > Hm. basilisk client (zellkonverter) failing on SPB on mac. I wish all these Python/conda bugs could be easy to fix for a change.

Aaron Lun (20:01:39): > Anyone else got a local mojave?

Peter Hickey (20:55:19): > dunno if helpful but for me on catalina, R CMD check (viadevtools::check()) errors out with > > Error: R CMD check found WARNINGs > Execution halted > > Exited with status 1. > - File (Plain Text): 00check.log

Aaron Lun (20:55:53): > is there an error here?

Peter Hickey (20:56:10): > i think so but i don’t see where …

Aaron Lun (20:56:31): > says WARNING and NOTE AFAICT.

Peter Hickey (20:57:18): > but the ‘execution halted’ and ‘exited with status 1’? perhaps some warning is being promoted to an error

Peter Hickey (21:00:41): > inR_check_bin/RscriptandR_check_bin/R(attached) there might be some hints … i don’t know if this is specific to my laptop > > echo "'Rscript' should not be used without a path -- see par. 1.6 of the manual" > exit 1 > > > > echo "'R' should not be used without a path -- see par. 1.6 of the manual" > exit 1 > - File (Zip): zellkonverter.Rcheck.zip

Aaron Lun (21:02:59): > In any case, whatdevtools::checkthinks is wrong is kind of irrelevant because the BBS runs via the CLI anyway.

Peter Hickey (21:03:22): > i’m running there now

Peter Hickey (21:18:00): > Run asR CMD build zellkonverterandR CMD check zellkonverter_0.99.0.tar.gzon command line. > > $ tail /Users/Peter/GitHub/zellkonverter.Rcheck/00check.log > * checking package vignettes in 'inst/doc' ... OK > * checking running R code from vignettes ... NONE > 'zellkonverter.Rmd' using 'UTF-8'... OK > * checking re-building of vignette outputs ... OK > * checking PDF version of manual ... WARNING > LaTeX errors when creating PDF version. > This typically indicates Rd problems. > * checking PDF version of manual without hyperrefs or index ... ERROR > * DONE > Status: 1 ERROR, 1 WARNING, 1 NOTE > - File (Zip): zellkonverter.Rcheck.zip

Aaron Lun (21:18:17): > yes, that’s expected.

Aaron Lun (21:18:39): > So looks like the SPB error is not reproducible.

Peter Hickey (21:20:46): > ok

2020-07-14

Luke Zappia (06:06:38): > @Luke Zappia has joined the channel

2020-07-20

Dr Awala Fortune O. (02:18:41): > @Dr Awala Fortune O. has joined the channel

Dr Awala Fortune O. (02:19:55): > Hello happy to be part of the team.

2020-07-22

Aaron Lun (12:09:27): > Did we skip a build last night?

Lori Shepherd (12:26:42): > The builds are still running for the day – its still early for a report

Hervé Pagès (17:55:43): > Looks like tokay1 stopped building at some point last night hence no report today. It didn’t start building today either. Need to go on the machine see what’s going on. Could be related to the latest SPB report for snifter not showing any output:http://bioconductor.org/spb_reports/snifter_buildreport_20200721160231.html#tokay1_buildsrc_anchor

Hervé Pagès (18:25:30): > Ouch! Can’t even rdesktop to the machine

Aaron Lun (18:26:36): > Windows.

Alan O’C (18:29:23): > Seems to be an equivalent error message from 2018 pointing to line breaks in DESCRIPTION, though I don’t see how that’d break the serverhttps://github.com/Bioconductor/Contributions/issues/843. (then again, Windows)

Hervé Pagès (18:32:16): > “Graceful Power Off” doesn’t work either, gonna have to do a “Force Power Off”. Power meter indicates normal build activity until 12:30 am (EST) last night. Then the curve is flat at 180 Watts. That still a lot of power consumption for not doing anything useful. Yes, Windows…

Hervé Pagès (18:37:27): > “Force Power Off” worked. Power consumption is down to 0. Making some progress. Pressing “Power On” now…

Aaron Lun (18:38:10): > Windows: when you unplug the computer, but the machine is still running.

Aaron Lun (18:38:14): > Like one of those horror movies.

Hervé Pagès (18:38:36): > The undeads

Aaron Lun (18:38:52): > And then STEVE BALLMER CRAWLS OUT OF THE SCREEEN

Hervé Pagès (18:39:22): > yeah that movie was a good one. What was the name?

Aaron Lun (18:39:49): > the ring?

Alan O’C (18:41:25): > Ballmer was a real character. Maybe the reason Linux has such low uptake for desktop is that you rarely see a coked up Torvalds screaming about developers

Hervé Pagès (18:46:51): > tokay1 is up and running again. Power consumption is at 127 Watts. That sounds more reasonable (still a lot for a server that is just waiting to do something if you ask me). rdesktop sucessful. Yes!

Hervé Pagès (19:18:15): > So my impression by looking at several logs is that tokay1 was in such a messed up state that the Task Scheduler (another master piece of crap by Microsoft) was not even able to start tasks. Too many zombie processes I guess. Yes the software builds (and maybe the SPB, don’t know about this one) tend to leave a few zombie processes behind (for all kinds of reasons) so one of the tasks we have in the Task Scheduler is a daily reboot a couple hours before the beginning of the daily builds. This way we start on a clean slate every day. But now we know: too many zombies will prevent the automatic reboot! Yes, it’s one of those horror movies:scream:Number of zombie processes is typically between around 1 or 2 dozens but this time it seems we had a lot more so next question is how we ended up with so many of them. This one will require more time to investigate.

Hervé Pagès (19:22:08): > From the Task Scheduler logs: > > Task Scheduler did not launch task "\BBS\DAILY SERVER REBOOT" as it missed its schedule. Consider using the configuration option to start the task when available, if schedule is missed. > > :grin:

Alan O’C (19:24:56): > Wow, maybe you can pitch this as a new horror blockbuster (28 builds later…?). R has a habit of making zombies on Linux when parallel processes are killed or fail, could be related

Hervé Pagès (19:32:50): > Yes, parallel processes that stay alive after the job is done is a common source of zombie processes and the build system needs to deal with them on all platforms, not just on Windows. On the Linux and Mac builders it’s justkill -9 -1at the end of the builds. A few years ago I tried to find some equivalent to this for Windows but none of the solutions I found on internet would really work as expected. So we decided to use the BFH (Big Fat Hammer). Not elegant but I thought at least with this one I’m safe. Turns out I’m not.

Alan O’C (19:37:14): > Perhaps it’s time to submit a PR to Redmond:smile:

Hervé Pagès (19:43:19): > or perhaps it’s time we stop using Windows for serious scientific computing

Hervé Pagès (20:02:25) (in thread): > I’ve only seen the original one though (Japanese). Now IMDb tells me that the US remake happens in Seattle, with many shots in various interesting places around Seattle. So I guess I’m going to make an exception to my “no remakes, thanks!” rule.

Aaron Lun (21:00:55): > did we ever start?

Hervé Pagès (23:12:59): > Right. Meant to say perhaps it’s time we stoptryingto use Windows for serious scientific computing. Not working:pensive:

2020-07-23

Aaron Lun (20:21:44): > Looks like the BBS is not happy abouthttp://bioconductor.org/checkResults/devel/bioc-LATEST/PCAtools/merida1-install.htmlon Mac.

Aaron Lun (20:22:55): > It’s not even recompiling the C++ code that should be present in the package, whereas the Linux build clearly is doing some recompilation. Little wonder, then, that the downstream loading of the library fails with a stale binary.

Kevin Blighe (20:24:06): > @Kevin Blighe has joined the channel

2020-07-27

Hervé Pagès (02:10:50): > @Aaron LunAFAICT the Linux build is not doing some recompilation either, only linking.@Kevin BligheThat should give you a clue (hint: you’ve added object files to your package source tree – don’t do that!). Also the whole thing should be completely reproducible.

Aaron Lun (02:12:37): > oh damn. Thanks.

Aaron Lun (02:14:38): > didn’t even occur to me that those object files were commited

Shian Su (18:49:39): > @Shian Su has left the channel

2020-07-28

Nur M Shahir (13:07:14): > @Nur M Shahir has joined the channel

2020-07-31

Dan Bunis (12:58:08): > Hey team, I’ve noticed that I’m getting a build error localized to Tokay1 which is separate from the ones above. (I figured I’d wait til towards the end of#bioc2020because the Mac and Linux builds for my package have been fine, so it didn’t seem super high priority.) > > The issue: > AFAICT, there seems to be an error with the Cairo installation/configuration. > > Thebuild reportgives this error in each instance of the package’s examples or tests that attempt to use plotly to create interactive plots: > > Error: Showing hover.data works for ScatterPlot (with cells.use) (@test-ho > .onLoad failed in loadNamespace() for 'Cairo', details: > call: library.dynam("Cairo", pkgname, libname) > error: DLL 'Cairo' not found: maybe not installed for this architecture? >

Erick Cuevas (14:12:05): > @Erick Cuevas has joined the channel

2020-08-03

Lori Shepherd (09:06:51): > @Dan Buniswe’ll look into it. Thanks for reporting

Aaron Lun (20:20:13): > So. Uh. The dependency badges.

Martin Morgan (20:27:45) (in thread): > cryptic

Lori Shepherd (21:17:34) (in thread): > It’s been on my to do list. Little back logged.

Aaron Lun (21:17:49) (in thread): > :+1:

2020-08-04

Lori Shepherd (13:41:36): > @Dan BunisThanks for reporting. We have reinstalled Cairo so this should hopefully clear up on tomorrow’s report

Lori Shepherd (13:45:38) (in thread): > Found the issue. Working with herve on the solution.

Dan Bunis (14:16:57): > Awesome, thanks! I’ll keep an eye on the report and confirm that it goes away.

2020-08-05

Dan Bunis (17:20:00): > Error is gone, and windows binary is built & propagated:heavy_check_mark:

2020-08-07

Aaron Lun (12:29:31): > I’m curious to know what’s going wrong with the iSEEu build reports. I’m not getting the little circles and it’s been 2-3 days since we bumped the version:http://bioconductor.org/checkResults/devel/bioc-LATEST/iSEEu/

Alan O’C (13:07:06): > Not sure if this is the right place to report but the bioc docker image fails with gh-actions, sysreqs points at the wrong deb package: > > Package python-minimal is not available, but is referred to by another package. > This may mean that the package is missing, has been obsoleted, or > is only available from another source > However the following packages replace it: > python2-minimal >

Alan O’C (13:20:01) (in thread): > Not a major problem as you can just use regular ubuntu but presumably it might pop up for other packages

Aaron Lun (14:23:33): > Ah, circles are back.

Hervé Pagès (17:47:02) (in thread): > You might get better results by reporting this on the#containerschannel

Hervé Pagès (17:52:44): > @Aaron LunLooks like simpleSingleCell should suggest scRNAseq:https://master.bioconductor.org/checkResults/3.12/workflows-LATEST/simpleSingleCell/nebbiolo1-buildsrc.html

Aaron Lun (17:53:25): > Oh yeah, I forgot that was still going.

Aaron Lun (17:53:48): > @Vince Careyand I finally got the book building on Anvil. Wondering what the best next step is for this entire process.

Vince Carey (17:54:16): > Was there success building book on bioc builders?

Aaron Lun (17:54:29): > Once.

Aaron Lun (17:54:33): > Or maybe twice.

Aaron Lun (17:54:47): > Problem is that I always forget to bump the version and the builds are pretty infrequent.

Hervé Pagès (17:55:54): > workflows build: 2x a week. And if we want to go for dedicated books builds, it won’t be more than that either (or maybe it would be 1x a week only, depending on resources)

Vince Carey (17:56:59): > OK – at the moment I am looking at Book+AnVIL as a demonstration of how to use AnVIL for learning single-cell transcriptomics and less as a piece of infrastructure to maintain the book. But we can put more effort into the latter if necessary. Or if Herve is happy to do builds on bioc system, maybe just figure out how to pull relevant artifacts from there to keep the AnVIL workspace up to date.

Aaron Lun (17:58:05): > oh. I thought it was going to bethe thingfor book building.

Hervé Pagès (18:00:20): > I’d be happy to set up book builds. Just let me know what you guys want to do. BBS is designed around the notion of package though so books would need to be somehow wrapped inside packages for a graceful handling by the BBS code.

Vince Carey (18:07:28): > It is possible that we will propose to use AnVIL for maintaining the book. We’ve demonstrated that we can do it, but we haven’t set up a change management approach. The fact that it uses Bioc 3.12 pushes it outside anything that could be regarded as standard for AnVIL. But if we want to establish a longer-term game plan, we can do so. I plan to spend a few minutes in an AnVIL tech call getting people acquainted with the issues and value of the process.

Aaron Lun (18:07:54): > would be interested in hearing the outcome of that

Vince Carey (18:08:15): > @Aaron Lunhave you used AnVIL in any way? Do you have an account in it?

Aaron Lun (18:08:26): > nope and nope.

Vince Carey (18:10:16): > OK – I think coming up with use cases for students and AnVIL users will take some effort – I’ve shown how one can pick a chapter and do the computations there. “Plug in your data here” is a process that I think will be worth demonstrating … I shouldreadthe book to see how that is addressed…..

2020-08-08

Sean Davis (15:01:10): > Thanks to Vince’s inspiration, OSCA book is now available in the workshop system.http://workshop.bioc.cancerdatasci.org/ - File (PNG): image.png

Sean Davis (15:02:01): - File (PNG): image.png

Aaron Lun (17:57:54): > very nice

Aaron Lun (18:03:06): > @Hervé PagèsI will work on an autobump system for simpleSingleCell to trigger book building as frequently as possible. In the meantime, what are your thoughts on my comments inhttps://github.com/Bioconductor/Contributions/issues/1565#issuecomment-668327741?

Hervé Pagès (18:06:37): > Why do you need an autobump system? Things get build whether their version is bumped or not. > As for the snifter story: sounds good to me.

Aaron Lun (18:08:59): > I would need it to push the rebuilt tarball, right?

Aaron Lun (18:09:14): > Because I need to get the compiled book out of the built tarball so that I can post it up on GitHub pages.

Hervé Pagès (18:10:00): > Don’t you need this only when you make changes to it?

Aaron Lun (18:10:29): > yes, but it’s quite hard to remember to bump a completely different repo when I make changes to the book.

Aaron Lun (18:11:43): > The book itself is at Bioconductor/OrchestratingSingleCellAnalysis, while the “trojan horse” that I’m using to sneak it onto the BBS iis at MarioniLab/simpleSingleCell.

Aaron Lun (18:12:29): > it’s fine, I’ll just set up a cron job on my laptop that does this.

Hervé Pagès (18:16:59): > I see. Because of the trojan horse setup this means your autobump will also trigger propagation of a new workflow twice a week even though it has not changed. So let’s call the current trojan horse + autobump trick less than ideal. Sounds like one more reason to come up with dedicated book builds. Where do we stand on that?

Aaron Lun (18:20:28): > I think there might be a way of doing it without breaking out of the BBS’ package paradigm.

Aaron Lun (18:21:05): > This would involve another layer on top of the BBS for all books.

Aaron Lun (18:22:34): > Submitters would just specify a Github repo (or BioC-git, I’m happy for either) to this book layer, with the contract there being that a system should just be able to callbookdown::render_book()or something similar (maybeMake?) to compile the contents of the book.

Aaron Lun (18:23:34): > Inside the book layer, we would construct a trojan package in the same manner assimpleSingleCellis doing; this is what would be provided to the BBS.

Aaron Lun (18:24:16): > So the BBS continues to sees packages and does its stuff, and it doesn’t need to know any better; the book layer then retrieves the compiled book from the package tarball and posts it somewhere on the BioC website.

Aaron Lun (18:26:10): > Basically, the book layer would just automate what I’m doing right now to interface with the BBS. A hook on pushes could conceivably also trigger a bump in the trojan package.

2020-08-09

Hervé Pagès (00:24:28): > We can point BBS to any git repo that contains something already organized like an R package. In particular it’s important to have a DESCRIPTION file that describes deps, authorship, version, license, etc… So for example in the case of the deps the build system knows what packages to install before building the book.

Aaron Lun (00:28:37): > I won’t haveR/,man/orvignettes/, though.

Hervé Pagès (00:35:10): > You would only need to havevignettes/(like for workflows). Why couldn’t you have that?

Aaron Lun (00:37:07): > I could, but I don’t think bookdown has a vignettebuilder, so we would still have to do some kind ofMakefilehack: this is what the simpleSingleCell trojan is currnetly using.

Hervé Pagès (00:41:10): > Right and that’s ok. The big difference is that the package would be for the bookonlyand not a workflow package hiding a book.

Aaron Lun (00:42:21): > right. Well, simpleSingleCell’s scientific content has been superceded by the book, so it is kind of fitting that its final role is as a PoC for book compilation.

Hervé Pagès (00:45:45): > Also my understanding is that right now the Makefile in simpleSingleCell is actually usinggit cloneto pull the book from a different place which IMO is the real hack. What I’m proposing is that BBS builds a package that already contains the book so the Makefile wouldn’t need togit cloneanything.

Aaron Lun (01:01:50): > I don’t disagree, but I will have to think about how I would do that without baking in the deployment details into the main OSCA repo. It would be a last resort to actually make avignettes/directory inhttps://github.com/Bioconductor/OrchestratingSingleCellAnalysisand move all the Rmd files inside.

Hervé Pagès (01:04:14): > Up to you. Looks like you’ve been offered other options (AnVIL, GH Actions) so whatever you think is going to be simpler for you.

Aaron Lun (21:35:27): > Huzzah! It is done.https://github.com/LTLA/TrojanBookBuilder

Aaron Lun (21:59:57): > Example GHA athttps://github.com/MarioniLab/simpleSingleCell/blob/master/.github/workflows/transplant.yaml

Aaron Lun (22:00:52): > This autobumps and PRs upon change to the OSCA repo, which allows me to just go in and push to the BioC git.

2020-08-10

Aaron Lun (00:47:18): > Might actually change the PR to amasterpush, actually.

Vince Carey (08:44:52): > My reading of this is that Bioc’s BBS will compute the book and convey results to the main book site as its content evolves through authors’ pushes, is that right? I would like to keep the deployment on AnVIL in sync with this, so that we have not just the book content but also a platform where users can explore the code live.@Aaron Lundo you plan to manage the Dockerfile for a container sufficient to define the environment for this platform? Is it expected that bioconductor_docker:* would be sufficient?

Aaron Lun (11:16:42): > I think that would be the strategy, yes. It makes most sense for Bioconductor to build a Bioc book, unless it becomes too resource intensive.

Aaron Lun (13:31:55): > Ah, another wine!

Hervé Pagès (14:39:06): > Yeah, and a quite powerful one BTW so drink in moderation. nebbiolo1 is a new Linux builder. It’s at DFCI in Boston. It builds/checks all Bioconductor software packages in less than 7h vs 14h20min for malbec1 vs 16h for merida1 vs 17h for tokay1. And we’re only using half of its power for now (only 27 cpus out of 72). Thanks@Vince Careyfor the great purchase! Still need to install/tweak a few things to make all packages with special needs happy.

Aaron Lun (15:05:07): > holy crap

Aaron Lun (15:05:33): > maybe the book will also compile faster

Hervé Pagès (15:14:24): > I’m confident it will. But right now it errors on nebbiolo1 because of the unstated dep on scRNAseq:https://bioconductor.org/checkResults/3.12/workflows-LATEST/simpleSingleCell/nebbiolo1-buildsrc.html

Aaron Lun (15:15:04): > yes, the work I did above will take care of that.

Hervé Pagès (15:15:42): > Great! So I guess we’ll find out tomorrow how fast it can build the book.

2020-08-11

Hervé Pagès (13:00:17): > @Aaron LunHeads-up: Workflows builds are done (no report yet). I went ahead and checked the result for simpleSingleCell: it failed because it could not install rebook. Just installed it manually on malbec1 & nebbiolo1 and restarted the whole thing on nebbiolo1. Too late to restart on malbec1 so I’m just leaving it as is.

Aaron Lun (13:00:38): > oh damn, I forgot to remove rebook manually. Thanks.

Aaron Lun (13:00:45): > BRB

Aaron Lun (13:11:30): > actually, I might as well just submit rebook to Bioc. I’ll need it for the SingleR book anyway.

Hervé Pagès (13:16:29): > Failed again:

Hervé Pagès (13:16:29): > > * checking for file 'simpleSingleCell/DESCRIPTION' ... OK > * preparing 'simpleSingleCell': > * checking DESCRIPTION meta-information ... OK > * installing the package to build vignettes > * creating vignettes ... ERROR > --- re-building 'batch.Rmd' using rmarkdown > --- finished re-building 'batch.Rmd' > > --- re-building 'bigdata.Rmd' using rmarkdown > --- finished re-building 'bigdata.Rmd' > > --- re-building 'de.Rmd' using rmarkdown > Error: processing vignette 'de.Rmd' failed with diagnostics: > Scanner error: mapping values are not allowed in this context at line 12, column 8 > --- failed re-building 'de.Rmd' > > --- re-building 'doublets.Rmd' using rmarkdown > --- finished re-building 'doublets.Rmd' > > --- re-building 'intro.Rmd' using rmarkdown > --- finished re-building 'intro.Rmd' > > --- re-building 'misc.Rmd' using rmarkdown > Loading required package: dbplyr > adding rname '[ftp://ftp.ncbi.nlm.nih.gov/geo/series/GSE61nnn/GSE61533/suppl/GSE61533_HTSEQ_count_results.xls.gz](ftp://ftp.ncbi.nlm.nih.gov/geo/series/GSE61nnn/GSE61533/suppl/GSE61533_HTSEQ_count_results.xls.gz)' > Loading required package: R.oo > Loading required package: R.methodsS3 > R.methodsS3 v1.8.0 (2020-02-14 07:10:20 UTC) successfully loaded. See ?R.methodsS3 for help. > R.oo v1.23.0 successfully loaded. See ?R.oo for help. > > Attaching package: 'R.oo' > > The following object is masked from 'package:R.methodsS3': > > throw > > The following objects are masked from 'package:methods': > > getClasses, getMethods > > The following objects are masked from 'package:base': > > attach, detach, load, save > > R.utils v2.9.2 successfully loaded. See ?R.utils for help. > > Attaching package: 'R.utils' > > The following object is masked from 'package:utils': > > timestamp > > The following objects are masked from 'package:base': > > cat, commandArgs, getOption, inherits, isOpen, nullfile, parse, > warnings > > Loading required package: SummarizedExperiment > Loading required package: GenomicRanges > Loading required package: stats4 > Loading required package: BiocGenerics > Loading required package: parallel > > Attaching package: 'BiocGenerics' > > The following objects are masked from 'package:parallel': > > clusterApply, clusterApplyLB, clusterCall, clusterEvalQ, > clusterExport, clusterMap, parApply, parCapply, parLapply, > parLapplyLB, parRapply, parSapply, parSapplyLB > > The following objects are masked from 'package:stats': > > IQR, mad, sd, var, xtabs > > The following objects are masked from 'package:base': > > Filter, Find, Map, Position, Reduce, anyDuplicated, append, > as.data.frame, basename, cbind, colnames, dirname, do.call, > duplicated, eval, evalq, get, grep, grepl, intersect, is.unsorted, > lapply, mapply, match, mget, order, paste, pmax, pmax.int, pmin, > pmin.int, rank, rbind, rownames, sapply, setdiff, sort, table, > tapply, union, unique, unsplit, which.max, which.min > > Loading required package: S4Vectors > > Attaching package: 'S4Vectors' > > The following object is masked from 'package:base': > > expand.grid > > Loading required package: IRanges > > Attaching package: 'IRanges' > > The following object is masked from 'package:R.oo': > > trim > > Loading required package: GenomeInfoDb > Loading required package: Biobase > Welcome to Bioconductor > > Vignettes contain introductory material; view with > 'browseVignettes()'. To cite Bioconductor, see > 'citation("Biobase")', and for packages 'citation("pkgname")'. > > Loading required package: DelayedArray > Loading required package: Matrix > > Attaching package: 'Matrix' > > The following object is masked from 'package:S4Vectors': > > expand > > Loading required package: matrixStats > > Attaching package: 'matrixStats' > > The following objects are masked from 'package:Biobase': > > anyMissing, rowMedians > > > Attaching package: 'DelayedArray' > > The following objects are masked from 'package:matrixStats': > > colMaxs, colMins, colRanges, rowMaxs, rowMins, rowRanges > > The following objects are masked from 'package:base': > > aperm, apply, rowsum > > Loading required package: ggplot2 > --- finished re-building 'misc.Rmd' > > --- re-building 'multibatch.Rmd' using rmarkdown > --- finished re-building 'multibatch.Rmd' > > --- re-building 'qc.Rmd' using rmarkdown > --- finished re-building 'qc.Rmd' > > --- re-building 'reads.Rmd' using rmarkdown > --- finished re-building 'reads.Rmd' > > --- re-building 'spike.Rmd' using rmarkdown > --- finished re-building 'spike.Rmd' > > --- re-building 'tenx.Rmd' using rmarkdown > --- finished re-building 'tenx.Rmd' > > --- re-building 'umis.Rmd' using rmarkdown > --- finished re-building 'umis.Rmd' > > --- re-building 'var.Rmd' using rmarkdown > --- finished re-building 'var.Rmd' > > for x in book/about-the-contributors.Rmd book/bach-mammary.Rmd book/beyond-r-basics.Rmd book/bibliography.Rmd book/big-data.Rmd book/cell-annotation.Rmd book/cell-cycle.Rmd book/clustering.Rmd book/data-infrastructure.Rmd book/data-integration.Rmd book/doublet-detection.Rmd book/feature-selection.Rmd book/grun-hsc.Rmd book/grun-pancreas.Rmd book/hca-bone-marrow.Rmd book/index.Rmd book/interactive.Rmd book/interoperability.Rmd book/introduction.Rmd book/lawlor-pancreas.Rmd book/learning-r-and-bioconductor.Rmd book/lun-416b.Rmd book/marker-detection.Rmd book/merged-pancreas.Rmd book/muraro-pancreas.Rmd book/nestorowa-hsc.Rmd book/normalization.Rmd book/nuclei-analysis.Rmd book/overview.Rmd book/paul-hsc.Rmd book/pijuan-embryo.Rmd book/protein-abundance.Rmd book/quality-control.Rmd book/reduced-dimensions.Rmd book/repertoire-seq.Rmd book/sample-comparisons.Rmd book/segerstolpe-pancreas.Rmd book/tenx-filtered-pbmc3k-4k-8k.Rmd book/tenx-repertoire-pbmc8k.Rmd book/tenx-unfiltered-pbmc4k.Rmd book/trajectory.Rmd book/zeisel-brain.Rmd; do > /bin/sh: 1: Syntax error: end of file unexpected (expecting "done") > make: ***** [Makefile:5: compiled] Error 2 > Error in tools::buildVignettes(dir = ".", tangle = TRUE) : > running 'make' failed > Execution halted >

Aaron Lun (13:16:53): > hm.

Aaron Lun (13:17:42): > looking at it now

Hervé Pagès (13:17:45): > Can you reproduce? It fails after 40 sec. only on nebbiolo1

Aaron Lun (13:18:32): > ah damn. I know what happened.

Aaron Lun (13:18:48): > My backslashes didn’t properly escape themselves when I moved them.

Aaron Lun (13:42:10): > running locally - so far so good…

Aaron Lun (13:49:28): > Alright, looks to get to the book itself and start compiling.

Aaron Lun (13:55:00): > Okay, looks good. I think we can re-fire a run on nebbiolo.

Hervé Pagès (14:19:59): > done

Aaron Lun (14:20:12): > :+1:

Aaron Lun (14:20:18): > Let’s hope for the best.

Hervé Pagès (14:22:14): > :wine_glass::crossed_fingers:

Hervé Pagès (16:22:56): > New error after running for a little bit more than 1h30:https://bioconductor.org/checkResults/3.12/workflows-LATEST/simpleSingleCell/nebbiolo1-buildsrc.html

Aaron Lun (16:23:43): > hm.

Aaron Lun (16:23:48): > well, we got pretty far, so that’s something.

Aaron Lun (16:25:05): > Woah. Hold on. this is an error duringsessionInfo()!

Aaron Lun (16:25:30): > It’s literally the last thing that I do, you can see hownestorowa-hsc.Rmdhas reached 100%.

Aaron Lun (16:26:46): > I wonder; did it run into some other package builds? There’s a whole bunch of “missingDESCRIPTION” warnings as well.

Hervé Pagès (17:59:31): > That’s probably it. Was probably too late to restart the workflows builds so they clashed with the beginning of the software builds. That’s when all Bioconductor software packages get reinstalled. We should have better luck on Friday. Just a good reminder why we cannot stretch the workflows build window more than it is, unless we reorganize the weekly build schedule a little. Anyway the good news is that the darn thing builds in 1h30 on nebbiolo1 vs 4h on malbec1. Not bad:slightly_smiling_face:

Aaron Lun (18:00:25): > well, it didn’t get all the way to the crunchiest part, which is usually ~1 hour. I would say it was still at least 30 minutes before it got to the final chapter.

Aaron Lun (18:00:56): > But at least there is nothing that theoretically doesn’t work.

Hervé Pagès (18:02:18): > oh I thought you said we’ve reached 100%. > > But at least there is nothing that theoretically doesn’t work. > unless there’s something in the last chapter that goes wrong

Aaron Lun (18:02:38): > it was 100% for the nestorowa chapter only

Aaron Lun (18:03:02): > so the error occurred at the very end of the chapter

Aaron Lun (18:03:11): > which confused me because usually nothing happens there

Hervé Pagès (18:03:17): > ok so let me remove that stupid cowboy hat

Aaron Lun (18:03:47): > I would project that we’d be saving about 30 minutes to an hour based on its current behavior

Hervé Pagès (18:04:34): > done. So will it be > 2h or < 2h? Bets are open.

Aaron Lun (18:05:18): > if it finishes in under two hours I will renamerebooktoHerveWasRight.

Hervé Pagès (18:06:15): > are you sure? there are some witnesses here so you’d better keep your word

Aaron Lun (18:06:27): > it will never finish in two hours

Aaron Lun (18:06:34): > And don’t try to cache anything!

Hervé Pagès (18:07:09): > ha ha we’ll see. no need to cache anything, I can just edit the EllapsedTime

2020-08-13

Aaron Lun (10:51:16): > alright. Let’s see how it goes today.

Hervé Pagès (14:18:41): > Workflows are build on Tuesdays and Fridays.

Aaron Lun (14:19:02): > oh.

2020-08-18

Aaron Lun (16:40:15): > YES YES YES

Aaron Lun (16:40:19): > http://bioconductor.org/checkResults/devel/workflows-LATEST/simpleSingleCell/nebbiolo1-buildsrc.html

Aaron Lun (16:40:26): > And it took just under 3 hours

Aaron Lun (16:40:31): > so I don’t have to change the name of the package

Hervé Pagès (17:01:42): > It’s a shame. You’re missing a great opportunity to contribute a package with an amazing name.

Aaron Lun (17:03:45): > i’ve got to say, 3 hours is still pretty damn fast

Aaron Lun (17:04:00): > might even be faster than my laptop

2020-08-19

Aaron Lun (19:01:23): > OH YEAH

Aaron Lun (19:01:30): > automation FTW.

Hervé Pagès (19:23:31): > :thinking_face:??!

Aaron Lun (19:23:58): > Look at this beautiful thing:https://github.com/Bioconductor/OrchestratingSingleCellAnalysis-devel/actions/runs/215843728

Aaron Lun (19:24:20): > Scraped the BioC website for the link to the latest tarball and then unpacked it for deployment to GitHub pages.

Hervé Pagès (19:26:45): > Ah ok. Nice. I see now that the previous mysterious/cryptic comment was intended to trigger a reaction. You got me.

Aaron Lun (19:27:19): > Well, I was going to post a link but then I got distracted by some fires

Aaron Lun (19:27:24): > virtual and actual

Hervé Pagès (19:33:39): > Nice setup. I like that the book source are now included in simpleSingleCell instead of being pulled at eachR CMD build. So you have something that’s pretty good here and that will hopefully be easy to reuse by other books. Sounds like it’s going to be time to move (and rename) simpleSingleCell to some dedicated book builds.

Aaron Lun (19:34:16): > yeah. got another book that could get the same treatment so we can work out a system for that one as well.

Hervé Pagès (19:34:55): > sounds good

2020-08-20

Hervé Pagès (04:40:20): > I’ve set up the book builds and moved simpleSingleCell there:https://bioconductor.org/checkResults/devel/books-LATEST/They run on Tuesdays and Fridays, like the workflow builds (nebbiolo1 will be back on the next build report on Friday). Source tarballs end up inhttps://bioconductor.org/packages/devel/books/src/contrib/(seehttps://bioconductor.org/packages/devel/books/src/contrib/PACKAGES).

Lori Shepherd (10:18:10) (in thread): > finally figured out what was going on. You should see it updated within the next day or two in devel. Cheers

Aaron Lun (11:07:11) (in thread): > sweet

Aaron Lun (21:47:48): > Thanks to@Lori ShepherdI can see my sweet, sweet dependencies! Only 57 for scran, I think that’s almost halved from when we were still depending on scater.

Aaron Lun (22:23:09): > SingleR’s dependencies definitely halved now that I cut out the ExperimentHub-dependent functionality into celldex.

Aaron Lun (22:23:22): > this is great, my packages are going to be so lean.

2020-08-21

Hervé Pagès (13:08:51): > @Aaron LunsimpleSingleCell gets built twice right now, once as part of the workflow builds (https://bioconductor.org/checkResults/3.12/workflows-LATEST/), and another time as part of the book builds (https://bioconductor.org/checkResults/3.12/books-LATEST/). Can we sort this out by turning it into 2 separate packages, one for the workflow and one for the book? You can submit the new book package and we’ll let it go thru. Thanks!

Aaron Lun (13:09:56): > Sure. Guess we’ll just call it the “OrchestratingSingleCellAnalysis” package, nice and simple.

Aaron Lun (13:10:35): > Wonder if we should bang a “Book” on the end.

Aaron Lun (13:10:39): > Just to make it clear.

Hervé Pagès (13:10:46): > boring. Thought you were going to go for something like AaronOrchestratingSingleCellAnalysisBookTrojan

Hervé Pagès (13:11:18): > kidding aside, yes, Book suffix sounds good

Aaron Lun (13:11:33): > I’ll set that up tonight, I’ll probably miss the current build

Hervé Pagès (13:11:43): > no rush, thanks!

Martin Morgan (14:26:32): > Maybe addType: Bookto the DESCRIPTION file?

2020-08-22

Aaron Lun (01:30:24): > @Hervé Pagèsjust realized that it would be best if we gothttps://github.com/Bioconductor/Contributions/issues/1590through first, this would save me from having to clone rebook’s contents into the OSCABook package.

Hervé Pagès (21:16:06): > rebook is already installed on the book builders so no need to clone it into the OSCABook package.

2020-08-23

Aaron Lun (19:45:56): > the book has been stripped from simplesinglecell.

Aaron Lun (20:20:58): > I’m ready to go withhttps://github.com/LTLA/OrchestratingSingleCellAnalysisBookbut I wonder whether a better naming scheme would be in order to emphasize that this is just a deployment detail rather than the actual book.

Aaron Lun (20:23:12): > Like “Builder”, “Constructor”, “Compiler”, or something similar.

2020-08-24

Hervé Pagès (13:10:39): > how about “BookWrap”?

Aaron Lun (18:56:25): > Ended up going withOrchestratingSingleCellAnalysisWrapper, which is exactly what it is, I think.

Hervé Pagès (19:05:08): > sounds good

2020-08-25

Aaron Lun (00:03:16): > it is done,https://github.com/Bioconductor/Contributions/issues/1607

Hervé Pagès (14:09:13): > I just accepted it.

Aaron Lun (19:19:53): > :+1:

2020-08-26

Aaron Lun (12:24:44): > What is the color scheme for the dependency badges?

Aaron Lun (12:25:02): > I’m yet to see a green badge anywhere.

Lori Shepherd (12:32:56): > There is no green since its a guideline – it is blue for acceptable, yellow if over 80 percentile, red if over 95

Stephanie Hicks (21:22:14): > @Stephanie Hicks has joined the channel

2020-08-27

Hervé Pagès (00:42:52): > I wanted to give IE a chance on riesling1 (new Windows builder, runs Windows Server 2019): > > Content from the website listed below is being > blocked by the Internet Enhanced > Security Configuration: > > about:blank > > If you trust this website, you can lower security settings for > the site by adding it to the Trusted sites zone. If you know > this website is on your local intranet, review help for > instructions on adding the site to the local intranet zone > instead. > > Important: adding this website to the Trusted sites zone will lower the > security settings for all content from this web site for all > applications, including Internet Explorer. > > :rolling_on_the_floor_laughing:Back to using Firefox.

FelixErnst (04:10:20): > I haven’t given up hope, that this might be the last decade we “have” to install the IE…

Spencer Nystrom (11:58:11): > @Spencer Nystrom has joined the channel

2020-09-01

Aaron Lun (18:31:50): > @Luke Zappia@Alan O’C@Vince Careyignore your error messages, basilisk should stabilize over the next few days.

2020-09-02

Aaron Lun (13:18:27): > oh, windows.

2020-09-06

Bob Policastro (08:35:19): > @Bob Policastro has joined the channel

2020-09-08

Aaron Lun (00:03:32): > Do the long-tests have their own status badges? I’d like to include them in my wall:https://github.com/LTLA

Aaron Lun (00:45:40): > some badges for the OSCA book builds would also be nice.

Hervé Pagès (00:59:12): > Badges on package landing pages only. No landing pages, no badges.

Aaron Lun (01:28:27): > just noticed beachmat’s long tests are failing… on windows… ugh.

Hervé Pagès (02:19:25): > We could enable automatic email notifications, like we do for release software builds. Could be something the maintainer opts in via.BBSoptions.

Lori Shepherd (14:55:04): > The Bioconductor Release 3.12 Schedule has been posted and is tentatively scheduled for October 28th. Important deadlines can be found on the website and will be posted here and on the bioc-devel atr-project.orgmailing list. As a quick summary: > > October 2: New package submission deadline (packages will still need to pass the formal review process to be included). Package submitted after this date are not guaranteed to be reviewed; they will be reviewed time pending. > > October 13 Current Release 3.11 Freeze. No changes will be accepted to the Bioconductor 3.11 branch after this date. > > October 21 No API changes to devel 3.12. Also, newly submitted packages need to be accepted by this date to be included in the release > > October 23 Deadline for packages to be passing R CMD install/build/check. > > October 27 Devel 3.12 branch is frozen to create release branch. > > October 28 Release Bioconductor 3.12

Hervé Pagès (14:58:57): > After 2 years of interruption, we’re building the workflows on Windows again:https://bioconductor.org/checkResults/3.12/workflows-LATEST/

Hervé Pagès (14:59:27): > Also after 4 years of interruption the data-experiment builds are back on Windows:https://bioconductor.org/checkResults/3.12/data-experiment-LATEST/

Aaron Lun (18:01:05): > Looks like the book built athttps://bioconductor.org/checkResults/3.12/books-LATEST/OSCA/nebbiolo1-buildsrc.html

Aaron Lun (18:01:13): > Is there an automated deployment tohttp://bioconductor.org/books/devel/OSCA/? - Attachment (bioconductor.org): Orchestrating Single-Cell Analysis with Bioconductor > Or: how I learned to stop worrying and love the t-SNEs.

Hervé Pagès (18:01:35): > working on it

Aaron Lun (18:03:55): > It’s probably taking longer because I keep throwing in more content. Almost every chapter is single threaded, so perhaps we could try out some parallel makes oncehttps://github.com/Bioconductor/BiocFileCache/pull/27is resolved.

Hervé Pagès (18:12:18): > It’s ok. It still builds in a little bit more than 3h on nebbiolo1 which is fine (timeout limit is 5h for the books). malbec1 will go away soon. Building the OSCA book already generates some significant load peaks at the moment so maybe we should avoid increasing parallelization to avoid making these load peaks worse. Especially since the book builds are running at the same time as the workflow builds so already competing for resources. It’ll get worse when other books will come in.

Hervé Pagès (21:57:35) (in thread): > Done:http://bioconductor.org/books/devel/OSCA/Book deployment will now happen automatically after each successful build (granted the version was bumped).

2020-09-09

Aaron Lun (11:19:09) (in thread): > hurrah

Aaron Lun (11:19:35) (in thread): > now if we redirectosca-dev.bioconductor.org, we can delete one of these repositories.

Matt Stone (14:05:24): > @Matt Stone has joined the channel

2020-09-10

Hervé Pagès (12:41:10): > <!channel>Sorry I messed up. No devel build report today:https://bioconductor.org/checkResults/3.12/bioc-LATEST/Should be back tomorrow.

Hervé Pagès (17:35:19) (in thread): > Not sure who’s “we”. Who managesosca-dev.bioconductor.org?

Aaron Lun (17:44:48) (in thread): > The OrchestratingSingleCellAnalysis-devel repo has aCNAMEfile pointing to it, but I assume that some BioC-owned process is reading from the file and authorizing the redirection; otherwise I could just put anything in there.

Hervé Pagès (17:47:00) (in thread): > I don’t know about these settings. Who helped you setting this up?

Aaron Lun (17:47:49) (in thread): > Think it was@Martin Morgan, but I don’t actually know; this was in the book before I started editing it.

Hervé Pagès (17:48:48) (in thread): > Maybe ask in the#osca-bookchannel?

Martin Morgan (20:12:28) (in thread): > I’ll look into this. It’s a ‘CNAME’ record underbioconductor.orgthat redirects tohttps://bioconductor.github.ioand then GitHub redirects internally from there; the easiest thing to do is simply remove the CNAME record so thatosca-dev.bioconductor.orggoes… nowhere

Aaron Lun (20:52:20) (in thread): > couldn’t the record inbioconductor.orgmade to point to the book’s BBS URL? And then we could just delete the whole GH repo for the compiled devel book.

2020-09-11

Martin Morgan (04:24:08) (in thread): > no, CNAME doesn’t ‘do’ URLs, just foo.bar -> baz.bim; we’d have to add a virtual host to our apache2 configuration onbioconductor.org. osca-dev is a very recent addition, right? so let’s just turn off github pages on the GitHub repository. What’s the plan for the release version of the book? Will it also be repository-based after the October release?

Aaron Lun (11:24:47) (in thread): > I would say that the release would also be based on @hpages’ new system, so we will need redirection in one form or another for the much-more-widely usedosca.bioconductor.org. Perhaps the simplest solution would be to keep the GH repos and just show links to the new book location that people can click through.

2020-09-12

Aaron Lun (23:53:10) (in thread): > It is done.osca-dev.bioconductor.orgredirects to the book location. But it does mean that we cannot delete the GitHub repo.

2020-09-13

Hervé Pagès (15:18:05) (in thread): > How about displaying a simple page atosca-dev.bioconductor.orgthat tell people where the new location is instead of using an automatic redirect? That way they get a chance to update their bookmarks and maybe one day we can get rid ofosca-dev.bioconductor.org.

Aaron Lun (23:46:55) (in thread): > Alright, I made the redirection notice a lot more interesting.

Aaron Lun (23:47:18) (in thread): > osca.bioconductor.orgwill probably need to stick around for some time, though, as I’m pretty sure that link is published somewhere.

2020-09-14

Hervé Pagès (11:43:58) (in thread): > Looks good. I wish I could type that fast.

2020-09-21

Belinda Phipson (20:16:35): > @Belinda Phipson has joined the channel

2020-09-22

Aaron Lun (21:07:30): > scRNAseq is getting chronic failures on riesling; it looks like there’s something wrong with the EHub cache, based onhttp://bioconductor.org/checkResults/devel/data-experiment-LATEST/scRNAseq/riesling1-checksrc.html

Aaron Lun (21:08:26): > There are also issues like: > > could not allocate memory (0 Mb) in C function 'R_AllocStringBuffer' > > which can’t possibly be the fault of the package, I don’t do anything that devious. Seems like some of the RDS files might be corrupt.

Aaron Lun (21:09:26): > (and also, this year’s download stats aren’t showing up when I click the badge on the landing page.)

Hervé Pagès (21:45:55): > mmh, I’ve asked@Lori Shepherdto help with the EH cache issue.

Hervé Pagès (21:47:11): > TheR_AllocStringBufferfailure seems to be a 32-bit Windows failure only. Do you have access to a Windows system? Would be helpful if you could try to reproduce.

Hervé Pagès (21:47:55): > I’ve taken care of the issue with the data-experiment download stats. They should be back tomorrow.

Hervé Pagès (21:49:46): > BTW we probably had theR_AllocStringBufferissue for a long time. We’re only seeing it now because we’re checking the data-experiment packages on Windows again after a several-year break.

2020-09-24

Aaron Lun (12:55:16): > brutal - File (PNG): Screen Shot 2020-09-24 at 9.54.22 AM.png

FelixErnst (12:55:46): > yep. All GHA stoped working as well (I mean the Bioc related ones)

2020-09-25

Aaron Lun (11:56:50): > Builds are back!

2020-09-26

Hervé Pagès (01:31:25): > They were never gone, only a little bit more reddish than usual.

Aaron Lun (01:32:49): > my wall is still covered with blood though

Aaron Lun (01:33:26): > Need to wait for the badges to propagate

Hervé Pagès (01:33:31): > crime scene

Hervé Pagès (01:34:42): > I’m surprised the badges didn’t propagate yet. The last builds were done more than 12 hours ago.

Aaron Lun (01:35:24): > yup, for examplehttp://bioconductor.org/packages/3.12/bioc/html/beachmat.htmlis still showing a red error badge - Attachment (Bioconductor): beachmat (development version) > Provides a consistent C++ class interface for reading from and writing data to a variety of commonly used matrix types. Ordinary matrices and several sparse/dense Matrix classes are directly supported, third-party S4 classes may be supported by external linkage, while all other matrices are handled by DelayedArray block processing.

Aaron Lun (01:35:37): > usually seems to take a while, from past experience.

Hervé Pagès (01:37:23): > Weird because my understanding is that the landing pages are normally refreshed only a couple hours after the builds are done and packages are propagated.@Lori ShepherdIs this delay expected?

FelixErnst (07:00:06): > On release the builds depending on BiocStyle are still red. On devel the new version of BiocStyle did arrive in time, but not on release. So thats probably why the crime scene cleanup crew was delayed.

Lori Shepherd (08:06:51): > Hmm… That is weird. They should’ve updated. I’ll investigate

Spencer Nystrom (09:36:39): > Could it just be the browser caching the old badges? I’ve had this happen before.

2020-09-28

Lori Shepherd (08:44:55): > I see updated badges (at least when I checked beachmat it is green now?) – I’ll check into the times the badges are generated – the build times were adjusted recently and I never adjusted the badges accordingly. –

FelixErnst (09:31:14) (in thread): > I think the problem was that the result for the build started on the 24th was not posted/finished on Friday. On Saturday evening (CET) the result from the build started on the 25th was posted and everything went green.

Lori Shepherd (09:36:02) (in thread): > yes the badges are off the build report so if a new report didn’t finish/didn’t post than the badge would not be updated. I adjusted the crontab so the build status and platform availability badges should get updated closer to when the new build report is available.

FelixErnst (09:40:54) (in thread): > (The badges were really just a symptom, since the whole build didn’t finish).

2020-09-29

Hervé Pagès (20:08:20): > @Aaron LunNo book report today, sorry. I did something stupid. I see the following error on nebbiolo1 (after approx. 7000 sec.):

Hervé Pagès (20:08:20): > > | > |................................................................. | 92% > label: nuclei-contamination (with options) > List of 1 > $ fig.cap: chr "Percentage of counts in the nuclei of the 10X brain dataset that are attributed to contamination from the ambie"| *_truncated_* > > > ***** caught segfault ***** > address 0x55b3dc6a6ed8, cause 'memory not mapped' > > Traceback: > 1: FUN(x, ...) > 2: .blockApply2(x, FUN = FUN, ..., grid = grid, BPPARAM = BPPARAM, by.row = FALSE) > 3: colBlockApply(x, BPPARAM = BPPARAM, FUN = sum_row_counts, genes = genes, runs = runs) > 4: .sum_across_features(x, ids, subset.row = subset.row, subset.col = subset.col, average = average, BPPARAM = BPPARAM) > 5: .local(x, ...) > 6: sumCountsAcrossFeatures(y, features) > 7: sumCountsAcrossFeatures(y, features) > 8: controlAmbience(nuclei, ambient, features = is.mito, mode = "proportion") > 9: eval(expr, envir, enclos) > 10: eval(expr, envir, enclos) > 11: withVisible(eval(expr, envir, enclos)) > 12: withCallingHandlers(withVisible(eval(expr, envir, enclos)), warning = wHandler, error = eHandler, message = mHandler) > 13: handle(ev <- withCallingHandlers(withVisible(eval(expr, envir, enclos)), warning = wHandler, error = eHandler, message = mHandler)) > 14: timing_fn(handle(ev <- withCallingHandlers(withVisible(eval(expr, envir, enclos)), warning = wHandler, error = eHandler, message = mHandler))) > 15: evaluate_call(expr, parsed$src[[i]], envir = envir, enclos = enclos, debug = debug, last = i == length(out), use_try = stop_on_error != 2L, keep_warning = keep_warning, keep_message = keep_message, output_handler = output_handler, include_timing = include_timing) > 16: evaluate::evaluate(...) > 17: evaluate(code, envir = env, new_device = FALSE, keep_warning = !isFALSE(options$warning), keep_message = !isFALSE(options$message), stop_on_error = if (options$error && options$include) 0L else 2L, output_handler = knit_handlers(options$render, options)) > 18: in_dir(input_dir(), evaluate(code, envir = env, new_device = FALSE, keep_warning = !isFALSE(options$warning), keep_message = !isFALSE(options$message), stop_on_error = if (options$error && options$include) 0L else 2L, output_handler = knit_handlers(options$render, options))) > 19: block_exec(params) > 20: call_block(x) > 21: process_group.block(group) > 22: process_group(group) > 23: withCallingHandlers(if (tangle) process_tangle(group) else process_group(group), error = function(e) { setwd(wd) cat(res, sep = "\n", file = output %n% "") message("Quitting from lines ", paste(current_lines(i), collapse = "-"), " (", knit_concord$get("infile"), ") ") }) > 24: process_file(text, output) > 25: knitr::knit(knit_input, knit_output, envir = envir, quiet = quiet) > 26: (function (input, output_format = NULL, ... <truncated because line is VERY long> > 27: do.call(rmarkdown::render, c(args[1], readRDS(args[2]), list(run_pandoc = FALSE))) > 28: eval(quote({ args = commandArgs(TRUE) bookdown:::source_utf8(args[4]) out = do.call(rmarkdown::render, c(args[1], readRDS(args[2]), list(run_pandoc = FALSE))) bookdown:::source_utf8(args[5]) out_expected = xfun::with_ext(args[1], ".md") if (out != out_expected) { file.rename(out, out_expected) attributes(out_expected) = attributes(out) out = out_expected } if (file.exists(args[3])) { res = readRDS(args[3]) res[[args[1]]] = out saveRDS(res, args[3]) } else saveRDS(setNames(list(out), args[1]), args[3])}), new.env()) > 29: eval(quote({ args = commandArgs(TRUE) bookdown:::source_utf8(args[4]) out = do.call(rmarkdown::render, c(args[1], readRDS(args[2]), list(run_pandoc = FALSE))) bookdown:::source_utf8(args[5]) out_expected = xfun::with_ext(args[1], ".md") if (out != out_expected) { file.rename(out, out_expected) attributes(out_expected) = attributes(out) out = out_expected } if (file.exists(args[3])) { res = readRDS(args[3]) res[[args[1]]] = out saveRDS(res, args[3]) } else saveRDS(setNames(list(out), args[1]), args[3])}), new.env()) > 30: eval(expr, p) > 31: eval(expr, p) > 32: eval.parent(substitute(eval(quote(expr), envir))) > 33: local({ args = commandArgs(TRUE) bookdown:::source_utf8(args[4]) out = do.call(rmarkdown::render, c(args[1], readRDS(args[2]), list(run_pandoc = FALSE))) bookdown:::source_utf8(args[5]) out_expected = xfun::with_ext(args[1], ".md") if (out != out_expected) { file.rename(out, out_expected) attributes(out_expected) = attributes(out) out = out_expected } if (file.exists(args[3])) { res = readRDS(args[3]) res[[args[1]]] = out saveRDS(res, args[3]) } else saveRDS(setNames(list(out), args[1]), args[3])}) > An irrecoverable exception occurred. R is aborting now ... > Segmentation fault (core dumped) > Error in Rscript_render(f, render_args, render_meta, add1, add2) : > Failed to compile nuclei-analysis.Rmd > Calls: <Anonymous> -> render_new_session -> Rscript_render > Execution halted > make: ***** [Makefile:4: compiled] Error 1 > Error in tools::buildVignettes(dir = ".", tangle = TRUE) : > running 'make' failed > Execution halted >

Hervé Pagès (20:08:20): > There was an error on malbec1 too, after approx. 8000 sec., but I lost the log so don’t know if it was the same (given the timings it looks like it probably failed at the same place so chances are high that it was the same error). Hopefully we’ll have a full report on Friday:crossed_fingers:

Aaron Lun (20:11:40): > ah bum.

Aaron Lun (23:41:05): > okay, that particular segfault should be fixed.

2020-10-01

Aaron Lun (23:47:38): > I’ve resolved the test failures athttp://bioconductor.org/checkResults/devel/data-experiment-LATEST/scRNAseq/riesling1-checksrc.htmlbut I can’t see how the following is my fault: > > > sce <- MessmerESCData() > snapshotDate(): 2020-09-24 > see ?scRNAseq and browseVignettes('scRNAseq') for documentation > downloading 1 resources > retrieving 1 resource > loading from cache > see ?scRNAseq and browseVignettes('scRNAseq') for documentation > downloading 1 resources > retrieving 1 resource > Warning: download failed > hub path: '[https://experimenthub.bioconductor.org/fetch/3122](https://experimenthub.bioconductor.org/fetch/3122)' > cache resource: 'EH3106 : 3122' > reason: database is locked > Error: failed to load resource > name: EH3106 > title: Messmer ESC colData > reason: 1 resources failed to download > Execution halted >

Aaron Lun (23:48:12): > Seems like there’s a lock file floating around on Riesling.

2020-10-02

Lori Shepherd (08:15:26): > Thanks I’ll look into it after the builders finish there later this morning

2020-10-04

Aaron Lun (21:40:15): > I should add that the same problem affects MouseGastrulationData@Jonathan Griffiths

2020-10-05

Jonathan Griffiths (03:39:10): > @Jonathan Griffiths has joined the channel

2020-10-06

Aaron Lun (00:56:21): > Similar problem in chipseqDBData,http://bioconductor.org/checkResults/devel/data-experiment-LATEST/chipseqDBData/riesling1-buildsrc.html

Lori Shepherd (06:41:50): > When I went into the builder I could download resources manually and didn’t see any lock files. I’ll try again. Maybe windows had a harder time with the parallel/multiple accessing. I’ll take another look and see if I can come up with anything better.

Aaron Lun (11:09:13): > Yeah, I don’t know that it’s specifically a lock file, but you can see from the error that it’s definitely a hub/cache problem, not something on the package side.

Lori Shepherd (11:14:24): > yes agreed – just being tricky to diagnose but I am trying.

Hervé Pagès (14:26:37): > <!channel>Something went seriously wrong with the latest builds on malbec1 (https://bioconductor.org/checkResults/3.12/bioc-LATEST/) with 1000+ ofFatal error: cannot create 'R_TempDir'errors. This is currently under investigation. Please ignore these errors.

Hervé Pagès (17:05:11): > Oh! Looks like we have a 2nd book:https://bioconductor.org/checkResults/3.12/books-LATEST/Nice.

Martin Morgan (17:11:28) (in thread): > Yeah looks really great athttps://bioconductor.org/books/3.12/SingleRBook/;@Stephanie Hicksand others looking for a model, Aaron’s package can be cloned withgit clonehttps://git.bioconductor.org/packages/SingleRand might serve as a good template for other books… there’s a tag in DESCRIPTION file, and the vignettes/ directory has a particular structure including a Makefile, but otherwise it behaves like a standard package. It took about an hour to build on my Mac.

Hervé Pagès (17:14:57) (in thread): > Less than 32 min. on nebbiolo1!:rocket::wink:

Aaron Lun (17:35:07): > huzzah

Aaron Lun (17:35:15): > now i need to go figure out what broke in the OSCA book again

Sara Ballouz (23:52:37): > @Sara Ballouz has joined the channel

2020-10-07

FelixErnst (02:45:57) (in thread): > While you’re at it: Figure 6.3 is missing. I am not sure, if that is a problem on my end, but different browser, cache clearing didn’t solve it:https://osca.bioconductor.org/quality-control.html#quality-control-motivation - Attachment (osca.bioconductor.org): Chapter 6 Quality Control | Orchestrating Single-Cell Analysis with Bioconductor > Online companion to ‘Orchestrating Single-Cell Analysis with Bioconductor’ manuscript by the Bioconductor team.

Aaron Lun (02:47:21) (in thread): > What is this athttp://bioconductor.org/books/devel/OSCA/quality-control.html? - Attachment (bioconductor.org): Chapter 6 Quality Control | Orchestrating Single-Cell Analysis with Bioconductor > Or: how I learned to stop worrying and love the t-SNEs.

FelixErnst (02:48:03) (in thread): > Firgure 6.3 is missing. The image ist no loaded, but I just relaized, that this is GitHub hosted(?)

FelixErnst (02:48:32) (in thread): > I didn’t catch up with the recent changes. BioC-build != GitHub

FelixErnst (02:48:51) (in thread): > bioconductor.org/books!=osca.bioconductor.orgam I right?

Aaron Lun (02:49:15) (in thread): > that’s right, eventually the latter will redirect to the former in the next reelase

FelixErnst (02:50:02) (in thread): > So this image is missing from the GitHub build:https://osca.bioconductor.org/P2_W02.quality-control_files/figure-html/qc-mito-zeisel-1.png

Aaron Lun (04:23:11) (in thread): > fxied

Aaron Lun (12:49:36): > Oh bum. basilisk’s clients are failing on tokay1… all for unique reasons. > > Look at zellkonverter’s error trace, for example: > > vs2015_runtime-14.16 | 2.2 MB | | 0% WARNING conda.gateways.disk.delete:unlink_or_rename_to_trash(140): Could not remove or rename C:\Users\BIOCBU~1\AppData\Local\basilisk\11730A~1.18\0\pkgs\vs2015_runtime-14.16.27012-h30e32a0_2\concrt140.dll. Please remove this file manually (you may need to reboot to free file handles) > WARNING conda.gateways.disk.delete:unlink_or_rename_to_trash(140): Could not remove or rename C:\Users\BIOCBU~1\AppData\Local\basilisk\11730A~1.18\0\pkgs\vs2015_runtime-14.16.27012-h30e32a0_2\concrt140.dll. Please remove this file manually (you may need to reboot to free file handles) > > vs2015_runtime-14.16 | 2.2 MB | ########## | 100% > > setuptools-49.6.0 | 953 KB | | 0% ERROR: Encountered corrupt package tarball at C:\Users\BIOCBU~1\AppData\Local\basilisk\11730A~1.18\0\pkgs\setuptools-49.6.0-py37hc8dfbb8_1.tar.bz2. Conda has left it in place. Please report this to the maintainers of your package. For the defaults channel, please report to[https://github.com/continuumio/anaconda-issues](https://github.com/continuumio/anaconda-issues) > > @Hervé Pagèscould you do me a favor and nukeC:\Users\BIOCBU~1\AppData\Local\basiliskon tokay1? Hopefully it’s a sporadic failure with the download.

Aaron Lun (13:10:24): > Also happy to be told that tokay1 is being retired for riesling, and then I won’t worry about it.

Aaron Lun (13:33:19): > Actually. Come to think about it, I suspectconda installis not thread-safe.

Hervé Pagès (13:33:37): > Nuked. That was big (> 15GB), much bigger than on the other builders. There was an old stale version there: 0.99.65. Actually other builders (e.g. malbec1) also have it but even with that thebasiliskfolder is “only” 7.1G on malbec1. Some builders don’t have the stale version (e.g. merida1), only the latest version (1.1.18), and the folder is only 3.1GB there. Would be nice to have a robust mechanism that keeps thebasiliskfolder clean or some users are going to start having disk space problems:wink:

Aaron Lun (13:36:28): > There is… but only for the current BioC version, the mechanism won’t deletebasiliskdirectories from other versions (e.g., if the cache is serving multiple R installations with different BioC versions, it doesn’t know which ones are still in use). I don’t have a good solution for that.

Hervé Pagès (13:55:37): > Crazy idea: use a “last time used” timestamp mechanism. 1) Use one timestamp perx.y.zsubfolder in thebasiliskfolder (can’t have 2 folders with samex.ythere since you’re already taking care of replacing previous basilisk versions within a given Bioconductor release). 2) Update the version-specific timestamp each time the user loads basilisk. 3) When the user loads basilisk, check all the timestamps for all the versions currently installed. If one of them is more than 6 months old, ask them if they want to remove it. 4) Remember their answer for a certain amount of time and ask again after this amount of time has passed.

Aaron Lun (13:56:11): > yeah, I was thinking about a timestamp as well.

Aaron Lun (13:56:33): > I was going for a warning rather than an interactive prompt, but basically the same the idea.

Aaron Lun (13:56:40): > Maybe something for the next release.

Aaron Lun (13:59:58): > probably could just usefile.info’satime, actually.

Hervé Pagès (14:00:48): > I wouldn’t trust this too much on Windows though but maybe it’s ok

Stephanie Hicks (14:02:16) (in thread): > awesome! thanks@Martin Morgan@Hervé Pagès!

Hervé Pagès (14:05:21): > Some tools like a virus scan will artificially bump theatime. Or afind+grepby the user when they’re trying to locate stuff (I do this a lot). The basic problem is that “last time accessed” != “last time the package was loaded”.

Sean Davis (21:32:55): > @Sean Davis has left the channel

2020-10-08

Aaron Lun (19:02:42): > Anyway, Basilisk’s clients are operational again.

Aaron Lun (19:07:05): > the timestamping thing will be tricky to implement. Guess I’ll have to make another file.

Hervé Pagès (22:15:41): > yes tricky and hard to test (this is the kind of stuff that is hard to write unit tests for)

2020-10-09

FelixErnst (02:17:57): > Hi I have quite peculiar error on riesling1 on Win32-bit only:https://bioconductor.org/checkResults/devel/bioc-LATEST/tRNAdbImport/riesling1-checksrc.html. This is the third day in a row I checked. I am not sure, what might be going on there and what to do about it. Any ideas?

FelixErnst (02:27:53): > Another thing:https://bioconductor.org/checkResults/devel/bioc-LATEST/STATUS_DB.txtreports errors for some packages, for exampleRNAmodR.AlkAnilineSeq, whereashttps://bioconductor.org/checkResults/devel/bioc-LATEST/index.htmldoes not. They should be from the same build, shouldn’t they?

Hervé Pagès (03:24:41): > @FelixErnstThey are from the same builds and the 2 files are in agreement in my browser. Make sure your browser cache is not getting in the way. ALSO:bioconductor.orgis served by Amazon CloudFront and it seems that Amazon CloudFront also uses a cache mechanism. Since this time the caching is on the server side a forced reload of the page (e.g. with F5 or CTRL + F5) won’t make any difference. Sometimes it can take a long time before Amazon CloudFront actually “realizes” that a page has changed (w.r.t. the page from the previous builds) and serves the new page. This has bitten me in several occasions in the past. A real annoyance!

FelixErnst (03:34:27): > This is interesting. Does the result change for you, when you compare the redirects? > For me on two different machines, different browser, CTRL+R, and everything I could think of to get a genuine reload:https://bioconductor.org/checkResults/3.12/bioc-LATEST/STATUS_DB.txt!=https://bioconductor.org/checkResults/devel/bioc-LATEST/STATUS_DB.txton my end. And the same result is also returned byBiocPkgTools, when runningBiocPkgTools::problemPage(..., ver = "devel")vs.BiocPkgTools::problemPage(..., ver = "3.12")

FelixErnst (03:35:32): > But okay, I hope I will remember the grain of salt next time.

Hervé Pagès (03:52:05): > For mehttps://bioconductor.org/checkResults/3.12/bioc-LATEST/STATUS_DB.txtandhttps://bioconductor.org/checkResults/devel/bioc-LATEST/STATUS_DB.txtare the same (I downloaded them withwgetand compared them withdiff). This is expected. But I don’t think that Amazon CloudFront has a way to know that so it will cache both. And depending on how/when it “realizes” that one or the other has changed it will update the cached copy of one but not necessarily the other so you end up with this situation. The 2 files that are served will eventually become the same again but it might take a while. At the moment, the 2 files that are servedarethe same otherwise mywget+difftest would have told me otherwise so it must be a cache problem on your side.

Hervé Pagès (03:53:36): > Oh, but wait! You are in Europe so probably hitting a different Amazon CloudFront cache (that’s the whole point of using Amazon CloudFront in the 1st place). So not because the cache is in sync from me means that it is for you. arghhh!

FelixErnst (03:55:04): > Jop, thats probably the case and it just the txt file not for the html file, which got me thinking in the first place. Server-side caching is just some voodoo, I didn’t take into account

Jonathan Griffiths (04:04:37) (in thread): > Hi Lori, was this fixed server wide or only for specific packages? (or perhaps it’s not fixed quite yet?) I’m still getting the issue in MGDhttp://bioconductor.org/checkResults/devel/data-experiment-LATEST/MouseGastrulationData/riesling1-checksrc.html

Hervé Pagès (04:09:06): > @FelixErnstFor thetRNAdbImport::open_tdbID()error on riesling1: > > > open_tdbID("tdbD00000785") > Error in shell.exec(url) : > file association for '[http://trna.bioinf.uni-leipzig.de/DataOutput/Result?ID=tdbD00000785](http://trna.bioinf.uni-leipzig.de/DataOutput/Result?ID=tdbD00000785)' not available or invalid > > We have a similar error forBiocDockerManager::help()on the same machine: > > > BiocDockerManager::help() > Error in shell.exec(url) : > file association for '[https://hub.docker.com/r/bioconductor/bioconductor_docker](https://hub.docker.com/r/bioconductor/bioconductor_docker)' not available or invalid > > Things work fine on tokay1 & tokay2 and I don’t remember having to do anything special there. Only difference is that I installed Google Chrome on the tokays but on riesling1 I installed Firefox. Could it be that Google Chrome installer does a better job than Firefox at creating these file associations? Or maybe I missed a checkbox in Firefox’s installer. Let me check.

FelixErnst (04:12:04): > It also limited to the 32bit tests forBiocDockerManager

FelixErnst (04:12:16): > 64bit work fine

Hervé Pagès (04:18:35): > According to the build report, it’s not in the tests, it’s in the examples. And, also according to the report, the examples fail in 32-bit and 64-bit modes fortRNAdbImport::open_tdbID()andBiocDockerManager::help().

Hervé Pagès (04:20:43): > Problem is that when I go on riesling1, I can’t reproduce this. Seems to only happen in the context of the daily builds. In an interactive session on riesling1,tRNAdbImport::open_tdbID()andBiocDockerManager::help()work fine:thinking_face:

Hervé Pagès (04:30:03): > A fullR CMD check --force-multiarch tRNAdbImport_1.7.1.tar.gzorR CMD check --force-multiarch BiocDockerManager_1.1.5.tar.gzalso works with no problem. Ok time to try replacing Firefox with Google Chrome.

Hervé Pagès (04:44:41): > Done. Too late to make a difference on today’s report (Friday). Let’s keep an eye on Saturday’s report.

FelixErnst (04:45:00): > Thanks for looking into it

2020-10-10

Hervé Pagès (04:40:41): > @FelixErnstDidn’t work (riesling1 is superfast and builds are already done there). Except for the version of Windows Server (2012 for tokay1, 2019 for riesling1), everything is the same on the two builders so I would be tempted to conclude that the problem is specific to Windows Server 2019. Happens whenR CMD checkis run via the Task Scheduler and not when run interactively. Hard to know if this is a bug inbrowseURL()that only manifests itself under these particular conditions or if it’s a bug in Windows Server 2019, or if it’s both. I guess not many people runR CMD checkon packages containing calls tobrowseURL()under this setup, so not too surprisingly Googling around didn’t give me much results. Will be a tough one to debug and, unfortunately, I won’t have much time for that before the release. > Can I suggest that you and Nitesh (@Nitesh Turaga) wrap the problematic example in a\dontteststatement for now? Thanks!

FelixErnst (05:20:33) (in thread): > will do. Thanks for trying anyway

2020-10-11

Nitesh Turaga (19:28:49) (in thread): > Just trying to follow up here, what exactly is required of me?

Hervé Pagès (19:32:29) (in thread): > Wrap the call toBiocDockerManager::help()that is currently failing on reisling1 inside a\dontteststatement. Not required, only suggested.:wink:

2020-10-12

Steffen Neumann (08:17:34): > Hi, if someone needs to fix a bug in a BioC package that only occurs on an architecture he/she has not access to, can I recommend to create a new dummy submission and use the SPB checks until that issue is fixed, and then push the fixes to the real BioC sources ? We don’t want to push non-working code to the main repo.

Martin Morgan (09:51:49): > No, there is a human intervention and other step required before the SPB will build a package, this is not appropriate. What’s the problem with pushing changes to the devel branch anyway, since the package is not building anyway? (I would not want to debug this way under normal circumstances, of course…)

Charlotte Soneson (09:55:33): > It might help to test the package using GitHub Actions with the appropriate operating system. At least it will give results back quickly.

Alan O’C (10:03:52): > I’ve been noticing discrepancies between GHA and BBS recently, though that may be an issue with my GHA setup

Charlotte Soneson (10:11:52): > Yep, that can certainly happen (also within the Bioc builders, some packages fail only on one of the Windows server versions, or one of the Ubuntu versions:slightly_smiling_face:) - but in case the bug can be reproduced with GHA, it at least provides a more rapid-response troubleshooting environment.

Alan O’C (10:13:15): > Yep it’s of course an upgrade from my previous method for debugging Windows and Mac errors (praying to the R gods)

Kevin Rue-Albrecht (11:11:42): > @Kevin Rue-Albrecht has joined the channel

Kevin Rue-Albrecht (11:20:10): > Is anyone aware of any package on the build system that commitedrenvfiles (including the.Rprofilefile). > I’m curious whether the build system has any issue withrenvfiles when building and checking packages. > Practically, I’m writing up my n-th “make a package” workflow (https://github.com/kevinrue/BiocPackageSofwareTemplate+ GitHub wiki) , and I’m wondering what the build system thinks of a package that has an.Rprofilefile at the top level. > How would the build system behave? I don’t see any--vanillaoption for R CMD build, so I imagine it just ignores any.Rprofile?

Hervé Pagès (12:12:03): > I don’t think this would cause any problem. I thinkR CMD buildalways ignores.Rprofile, whether--vanillais used or not.

2020-10-13

Aaron Lun (02:06:27): > @Vince Carey@Luke Zappia@Kevin Rue-AlbrechtA bit late in the day, but I just switched to the latest version of miniconda in basilisk. Hopefully nothing breaks, but if it does, I’ll just switch back.

Nitesh Turaga (14:22:41) (in thread): > Got it…

Hervé Pagès (18:46:23) (in thread): > @FelixErnst@Nitesh TuragaAlternatively you could precede the call toutils::browseURL()withif (interactive()). Note that this is whathttr::BROWSE()does.

Aaron Lun (20:06:41): > Argh. Book failed for three different reasons on all three builders.

Aaron Lun (20:43:00): > Ehub failure on malbec. Weird S4 failure on nebiblio (probably the only valid error). Who knows what’s happening on riesling.

Aaron Lun (20:57:27) (in thread): > Dammit, missed@Alan O’C.

Hervé Pagès (21:41:06): > We updated R to R 4.0.3 yesterday on malbec1 and nebbiolo1 (and today on a few other builders), We also flushed the data caches (Ahub, Ehub, and BFC). I suspect the Ehub failures are due to clashing concurrent writes to the cache, again. They are becoming a lot more frequent these days, probably because a lot more code is relying on Ehub, Ahub, and BFC.@Lori ShepherdAre there plans to make BFC safe to concurrent writes?

Hervé Pagès (21:44:15): > On riesling1 the error isError in Biocpkg("scRNAseq") : could not find function "Biocpkg". I think it’s the same since we’ve started to build the book on this machine.

Aaron Lun (21:45:46): > Hm. Funny it’s never shown up in the other builders.

Aaron Lun (21:46:00): > Maybe because the other builders use forks.

Hervé Pagès (21:46:04): > Funny but very consistent

Aaron Lun (21:51:46): > I probably never noticed it because it never affected the propagation

Hervé Pagès (21:56:46): > Right, we only propagate whatever gets built on malbec1. We build on the other machines just for fun (and QA).

Aaron Lun (22:21:06): > Hmm… I don’t think thatattachNamespaceworks on Windows like how I thought it worked.

Aaron Lun (22:33:43): > The nebiblio error is the only valid one, but it seems to suggest that the book is being compiled with scRNAseq version below 2.3.16; surely the propagation shouldn’t take a day…

2020-10-14

Hervé Pagès (00:17:38): > FWIW you can see all the packages + versions that were present on a given builder at the time the builds started by clicking on one of the links in the******Installed pkgs ****column in the top table of the report. For nebbiolo1:https://bioconductor.org/checkResults/3.12/books-LATEST/nebbiolo1-R-instpkgs.htmlSo yes, it’s still scRNAseq 2.3.15. > Propagation of scRNAseq 2.3.16 happened on Monday afternoon so normallyinstall.packages()should have been able to see it in thehttps://bioconductor.org/packages/3.12/data/experimentrepo when the book builds started this morning. Not sure why it didn’t so I’m going to blame the Amazon CloudFront again:grin:Right now I can see that scRNAseq 2.3.16 made its way to nebbiolo1. Looks like the software builds that are currently running picked it up at the beginning of the run. They probably picked it up on the other machines too.

Lori Shepherd (07:46:54) (in thread): > Yes@Aaron LunI think has PR in for it too. I’ll try to look at it tomorrow or Friday.

Aaron Lun (15:35:04) (in thread): > Looks like everyone’s still green?

Dan Bunis (15:36:21): > I’m not sure if recent changes to the build system might be a cause, but I’m getting a build error resulting from a CRAN packagepolyclipbeing unavailable: > Build report:http://bioconductor.org/checkResults/devel/bioc-LATEST/dittoSeq/tokay1-buildsrc.html > > Quitting from lines 103-112 (dittoSeq.Rmd) > Error: processing vignette 'dittoSeq.Rmd' failed with diagnostics: > package or namespace load failed for 'Seurat' in loadNamespace(i, c(lib.loc, .libPaths()), versionCheck = vI[[i]]): > there is no package called 'polyclip' > --- failed re-building 'dittoSeq.Rmd' > > Is this something that would clear up over time, or might it be a real issue?

Alan O’C (15:36:50) (in thread): > Yep, had some errors yesterday but all good now

Hervé Pagès (17:59:39): > We’ve updated R to R 4.0.3 on tokay1 (and on other builders) yesterday . After an R update, the build system needs to reinstall all deps on all the builders. Don’t know why polyclip failed to reinstall on tokay1, and I can’t go on the machine now to check (Windows builders are too fragile to let me do that). FWIW the reinstallation process had other serious hiccups, notably on nebbiolo1 where a few big data packages didn’t get reinstalled correctly. The build system will try again to reinstall the missing stuff at the beginning of each run but it might take a couple of days before the situation returns to normal.

2020-10-15

Luke Zappia (03:29:41) (in thread): > zellkonverteris now failing on everything. It looks like there are conflicts in the environment so maybe my problem. Didn’t happen before though so possibly related?

Nitesh Turaga (08:37:57) (in thread): > Done.

Aaron Lun (11:08:43) (in thread): > That looks brutal. Let me try.

Aaron Lun (11:11:31) (in thread): > The oddest thing is that velociraptor is fine, despite depending on zellkonverter.

Luke Zappia (11:12:43) (in thread): > Yes, that is weird… Unless it’s still using a prebuilt version?

Aaron Lun (11:13:20) (in thread): > I don’t think so, a basilisk update will destroy all caches for all downstream packages.

Aaron Lun (11:13:35) (in thread): > Current suspect is some older versions of numpy and pandas

Luke Zappia (11:14:33) (in thread): > I thought it might be something like that but I haven’t had a chance to go through and check yet

Aaron Lun (11:17:03) (in thread): > I can repro it

Aaron Lun (11:26:24) (in thread): > This eventually worked for me: > > .AnnDataDependencies <- c( > "anndata==0.7.4", > "h5py==2.10.0", > "hdf5==1.10.5", > "natsort==7.0.1", > "numpy==1.19.1", > "packaging==20.4", > "pandas==1.1.2", > "scipy==1.5.2", > "sqlite==3.33.0" > ) > > Seemssqlitewas the killer.

Luke Zappia (11:35:45) (in thread): > :+1:Thanks!

Hervé Pagès (13:14:00): > <!channel>Sorry no new build report today.

Melanie Loth (17:13:15): > @Melanie Loth has joined the channel

Jianhai Zhang (17:27:19): > @Jianhai Zhang has joined the channel

Koen Van den Berge (19:55:45): > @Koen Van den Berge has joined the channel

2020-10-16

Patrick Kimes (13:45:40): > @Patrick Kimes has joined the channel

Hervé Pagès (17:24:12): > @Aaron LunSo I don’t know what’s the “damn forking problem that I’ve been banging my head against for days” that you mention earlier in#delayed_arrayso I’m not really sure how I can help you with that. Is this about the book failing on riesling1? If that’s the case unfortunately I won’t have time to go on riesling1 to troubleshoot this, at least not until after the release. However what I can do is schedule the book builds to run 3 times a week (Mon, Wed, Fri) instead of 2 (Tue, Fri), if that helps.

Aaron Lun (17:24:34): > No, it’s unrelated.

Aaron Lun (17:24:52): > The book is fine. I was just trying to use our DA machinery to process 1 M cells on our server. Wasn’t pretty.

Aaron Lun (17:25:29): > I’ve gone through MCParam, SnowParam, and BTParam, and each one has failed in one way or another.

Hervé Pagès (17:25:58): > ah ok, I guess there’s not much I can do then

Hervé Pagès (17:26:13): > at least not without more info

Aaron Lun (17:26:30): > I had high hopes for SnowParam, but then it just fails with the process getting killed inexplicably.

Hervé Pagès (17:26:54): > I mean I’m in no way a BiocParallel expert

Hervé Pagès (17:27:59): > so I’m probably not the best person to help here

Aaron Lun (17:28:55): > BTParam fails in a cruel pincer move. On release, it fails because DA still usesbpiteratethere, and that fails withhttps://github.com/Bioconductor/BiocParallel/issues/107. On devel, it fails due to our previously discussed naming issuehttps://github.com/Bioconductor/BiocParallel/issues/123.

Hervé Pagès (17:29:42): > sounds like the gods are against us

Aaron Lun (17:30:05): > hence the head -> table action that has been happening for the last half of the week.

Hervé Pagès (17:30:39): > you should stop now, won’t help

Hervé Pagès (17:31:04): > don’t do it on the new screen though

Hervé Pagès (17:31:09): > would be a shame

Hervé Pagès (17:31:29): > BTW I just received mine

Hervé Pagès (17:31:34): > still in the box

Aaron Lun (17:32:38): > you should be tearing that box apart

Aaron Lun (17:32:42): > that’s what I did

Aaron Lun (17:32:54): > just dropped all my shopping, didn’t even put stuff in the fridge

Hervé Pagès (17:33:28): > but that was a few days ago. is the stuff in the fridge now?

Hervé Pagès (17:34:30): > ok I’m going to carefully open that box now. wouldn’t want to damage any pixel

Hervé Pagès (18:39:49): > It’s on my desk. Looks great! I was showing this videohttps://www.youtube.com/watch?v=BHACKCNDMW8on the new screen to wife and daughter and now they both want a new screen. Such a beginner mistake! - Attachment (YouTube): 3 Hours of Amazing Nature Scenery & Relaxing Music for Stress Relief.

Jianhai Zhang (21:31:25): > I bumped the package version, and run git push upstream master, but no email notification of Received a valid push ongit.bioconductor.org, etc. Anyone has the same issue?

Hervé Pagès (21:44:42): > Bumping the version to trigger a build by the SPB was only during the submission process. Your package has been accepted now and added to thedaily builds. On acceptance you’ve received some information about this. These builds run everyday at a fixed time on all Bioconductor packages. They start around 2:30 pm EST. They are not triggered by maintainers bumping the version of their package or not. You will no longer receive email notifications when you bump the version of your package or when the builds start or when they finish. The report is normally published every day around noon-1pm EST. You might receive an automatic email twice a week (Mondays and Thursdays) if your package has an ERROR or TIMEOUT on malbec1, our primary builder. > Note that when a package passes BUILD and CHECK with no ERROR or TIMEOUT, it is then published (i.e. made available viaBiocManager::install()) but we won’t replace an already published version of the package unless the version was bumped. We call this process “propagation”. You can see the version of the package that is currently published by looking at itslanding page:http://bioconductor.org/packages/spatialHeatmapHope this helps. - Attachment (Bioconductor): spatialHeatmap (development version) > The spatialHeatmap package provides functionalities for visualizing cell-, tissue- and organ-specific data of biological assays by coloring the corresponding spatial features defined in anatomical images according to a numeric color key.

Jianhai Zhang (21:51:32): > Then after I made changes to the package, in order to publish these changes on bioc, I should keep the same version on the landing page, rather than increase the pacakge version, is it right?

Hervé Pagès (21:56:20): > If you want your change to propagate, you need to bump the version.

Jianhai Zhang (21:57:56): > After I bumped the version, I can only see the changes on bioc after the next nigthly build is completed?

Hervé Pagès (22:02:41): > You will always see the result of your changes on next day’s build report, whether you bumped the version or not (as long as you made your changes before the cut-off time of the builds, around 2:30 pm EST, otherwise you’ll have to wait one more day). Bumping the version controls propagation, not whether we’re going to build your package or not. We build all packages every day (or I should say night, maybe, depending on which part of the world you are), whether you make changes to it or not, whether you~build~bump its version or not.

Jianhai Zhang (22:06:05): > Does the “propagation” only relate to the version change? Any other effects could be indcued?

Hervé Pagès (22:14:10): > Not sure what you are asking. As I explained above, propagation is the process of making your package available viaBiocManager::install()so people can install it (this works by putting it in package repository whereBiocManager::install()can “see” it), and generating its landing page. That’s the main effect. Don’t know what other effects you’re thinking of.

Jianhai Zhang (22:16:45): > I got the answer, and learned a lot. Thank you so much!

Jianhai Zhang (22:20:51): > Before I consulted you on slack. I bumped the version from 15 to 17 by mistake, then bumped again from 17 to 16. Can I keep 16 for the release? Next time I will remember to bump to 18 (from 16) if version bumping is needed.

Jianhai Zhang (22:21:52): > I remember you said the version can only increase not decrease.

Jianhai Zhang (22:22:47): > Or do I need to bump from 16 to 18 now?

Hervé Pagès (22:41:44): > The version that is currently published is 0.99.15. Versions 0.99.17 or 0.99.16 never made it. Yes versions shouldalwaysgo up. So you should bump to 0.99.18. Note that in this particular case, since the published version is 0.99.15, it doesn’t really matter if you set the version to 0.99.16, 0.99.17, or 0.99.18, as long as you set it to something that is > 0.99.15. This way,if it passes BUILD and CHECK, it will be allowed to propagate. Also note that on release daywewill bump its version to 1.0.0.

Jianhai Zhang (22:47:37): > Understand, thanks!

2020-10-18

Jianhai Zhang (16:45:14): > I tested my package on Windows Server 2019 evaluation (https://www.microsoft.com/en-US/evalcenter/evaluate-windows-server-2019?filetype=ISO), and there are no such errors raised onhttps://bioconductor.org/checkResults/3.12/bioc-LATEST/spatialHeatmap/tokay1-buildsrc.html, do you think Windows Server 2019 evaluation is equivalent with Windows Server 2019 Standard?@Hervé Pagès - Attachment (microsoft.com): Try Windows Server 2019 on Microsoft Evaluation Center > Full-featured Windows Server 2019 product evaluation available for download.

Jianhai Zhang (16:46:42): > I made necessary changes in version 0.99.16 on Friday. When will the 16 version be published on the devel branch?

Jianhai Zhang (16:48:06): > Now the landing page is 0.99.15

Jianhai Zhang (16:49:24): > Sorry, forget the version change. Now the landing page is 0.99.16.

Jianhai Zhang (16:50:58): > but the pdf manual is Version 0.99.15.

Hervé Pagès (17:03:57): > You have an “invalid character” error and we see it on our 2 Windows builders, Windows Server 2012 and Windows Server 2019. I suspect you’re using some characters that are not compatible with the default charset we use on the builders. You should stick to ASCII or UTF8 characters in your code for cross-platform/cross-region compatibility.

2020-10-19

Aaron Lun (01:26:33): > I hypothesize that the OSCA build failure on windows is caused by 2 problems. The first is thatknitris somehow, on Windows, storing its cache in a different location than whatrebookexpects. This causesrebookto miss the fact thatzeisel-brain.Rmdhad already been compiled inquality-control.Rmd, and doesn’t need to be compiled again innormalization.Rmd. > > Ordinarily a recompilation wouldn’t be a problem (beyond the loss of efficiency from redundant work), but this exposes a second problem. Remember thezeisel-brain.Rmdcache that generated byquality-control.Rmd? Well, it’s recognized byknitrupon recompilation, but only in a half-hearted way; it does not re-evaluate the chunk that loads inrebook(and by extension,BiocStyle), leading to the error whereBiocpkgis not found. This happens on Linux, but apparently not on Windows, god knows why.

Aaron Lun (01:36:58): > I’ll put in a patch to fix the second problem, but the first one is the real fix.

Jonathan Griffiths (04:10:21): > Hi - I’ve had a few issues with the Bioc build system for my package to do with locked or readonly databases, first on the Windows machines, and nowon a Linux one. This doesn’t seem to be a widespread issue (in terms of affecting other packages) - is this something that I can expect to shake out in the wash in the next run? > > I’d usually be happy to wait for another run of the package builder but we are getting a bit close to the release deadlines:grimacing:

Elana Fertig (10:06:22): > @Elana Fertig has joined the channel

Elana Fertig (10:08:59): > We are seeing a Riesling specific compilation error in CoGAPS that@Dirk Eddelbuettelhas traced to a likely memory issue as described on#random@Alan O’Crecommended we check in about the amount of memory on that builder. We aren’t sure how to fix the compiler specific error@Melanie Loth

Hervé Pagès (11:51:08) (in thread): > Sounds good. What’s the real fix?

Aaron Lun (11:52:28) (in thread): > I dunno, one would have to sit down at a windows machine to try to repro it to figure out where the cache is actually going.

Aaron Lun (11:53:10) (in thread): > I guess I could try with my VM, but it screams in pain when I even just try to type into the terminal.

Hervé Pagès (11:54:18) (in thread): > Isn’t there a short code snippet that can be run to figure out where the cache is actually going?

Aaron Lun (11:55:45) (in thread): > Yes, the?extractCachedexample inrebookwould be the prime candidate. I ran it yesterday on my VM and it didn’t complain - it also doesn’t complain on the BBS.

Hervé Pagès (12:00:36) (in thread): > Yes I can run theextractCachedexample on riesling1 with no problems. What’s next?

Lori Shepherd (12:05:32) (in thread): > We are working on a thread safety mechanism thanks to@Aaron Lunthat should be implemented shortly to help alleviate these types of errors. I would expect it to clear up on a future build and do not think you need to take any action at this time. sorry for the inconvenience

Aaron Lun (12:05:42) (in thread): > Hm. The shortest path to reliably reproducing the problem is to: > 1. Clone the OSCA book on riesling. > 2. Wander over tovignettes/bookand runbookdown::render_book("index.Rmd"). This should only run for about 10-15 minutes before failing at the error. > 3. Move everything out of_bookdown_filesinto the same directory as the Rmarkdown files. (Forgot that bookdown does this. V. annoying.) > 4. In the same directory, see if you can runextractCached("zeisel-brain.Rmd", "quality-control", "sce.zeisel"), which should reproduce the error. If we get to this point, we can start drilling down into whatextractCachedis believing vs. reality.

Hervé Pagès (12:28:51): > My googling indicates that this is a bug in the compiler, specific to the x64 arch:https://gcc.gnu.org/bugzilla/show_bug.cgi?id=65782riesling1 has 128GB of RAM so lack of memory is unlikely to be the problem here. The bug is fixed in recent versions ofgccbut on Windows we’re stuck with whatever version ofgccis shipped with Rtools4. Some people have reported being able to work around the problem by using compiler flag-fno-asynchronous-unwind-tables. Let me see if that helps.

Hervé Pagès (13:11:06): > The workaround allows compilation of the problematic file (math/Math.cpp) but then linking fails with hundreds of errors like this: > > C:/rtools40/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/8.3.0/../../../../x86_64-w64-mingw32/bin/ld.exe: math/VectorMath.o:VectorMath.cpp:(.text. > startup+0x53): undefined reference to `_Unwind_SjLj_Register' > > Other people have reported that the “invalid register for .seh_savexmm” error only happens with some compiler optimization levels. Let me see if changing the-Olevel helps.

Hervé Pagès (13:27:49): > mmh.. it works but unfortunately I have to completely disable optimization since using-O1is not enough. This might slow down your code significantly. Also I’m not sure where to put-O0in yoursrc/Makevars.winfile. To override the default (-O2),-O0needs to be placedafter-O2in the compilation commands. However, if I append-O0toPKG_CXXFLAGS,R CMD INSTALLwill place itbeforeso it’ll be ignored. > Someone knows where to put-O0so it gets inserted after-O2in the compiler commands?

Hervé Pagès (13:36:07): > Another option is that you ignore the problem for now. riesling1 is only a backup for tokay1 at the moment and the build results on riesling1 don’t impact propagation so won’t affect the release of the package. However, the plan is to move riesling1 to the 3.13 builds in a couple of weeks and to eventually make it the primary Windows builder for BioC 3.13. When this happens we’ll need to find a good solution to the “invalid register for .seh_savexmm” error.

Elana Fertig (13:40:34): > this seems like it shoudl be an issue for the bioc compiler vs packages overall? We’re trying to use a boost reliance which shoudl be an issue beyond CoGAPS, correct?

Hervé Pagès (13:54:26): > AFAICS CoGAPS is the only affected package. But wait… based on@Dirk Eddelbuettelcomment in#randomI thought you were depending on his BH package but you’re not. Instead you’ve included your own copy of the boost headers in the package. mmh that is not and ideal situation. This might explain why the other Bioconductor packages that use BH are not affected. Any reason why you’re not using BH? This might make the issue go away.

Hervé Pagès (14:39:26) (in thread): > ok, I’ve reached step 4: > > > extractCached("zeisel-brain.Rmd", "quality-control", "sce.zeisel") > # > # > # processing file: zeisel-brain.Rmd > # > # | > # | | 0% > # | > # |. | 2% > # ordinary text without R code > # > # > # | > # |... | 4% > # label: unref-setup (with options) > # List of 2 > # $ echo : logi FALSE > # $ results: chr "asis" > # > # > # | > # |.... | 6% > # inline R code fragments > # > # Quitting from lines 8-19 (zeisel-brain.Rmd) > # > # Error in Biocpkg("scRNAseq") : could not find function "Biocpkg" > > Error in compileChapter(path) : failed to compile 'zeisel-brain.Rmd' > > What’s next?

Aaron Lun (15:05:23) (in thread): > Alright. Hm.

Aaron Lun (15:06:12) (in thread): > Do you see azeisel-brain_cachedirectory in the current WD?

Aaron Lun (15:07:04) (in thread): > And does it have ahtml/subdirectory?

Hervé Pagès (15:08:23) (in thread): > Had to leave riesling1 and to disconnect VPN. Let me reconnect…

Hervé Pagès (15:09:00) (in thread): > Enabling VPN screws my Spotify streaming… very annoying!

Hervé Pagès (15:14:38) (in thread): > > Do you see azeisel-brain_cachedirectory in the current WD? > Yes, I copied it from_bookdown_filesas per your previous instructions. > > does it have ahtml/subdirectory? > Yes. That’s the only thing there. Contains a bunch.RData,.rdb, and.rdxfiles. > What’s next?

Aaron Lun (15:15:48) (in thread): > INTERESTING. Okay. Well. can youdebug(extractCached)and step through for the first few lines until: > > if (!file.exists(cache_path)) { > compileChapter(path) > } > > It should not be entering this block, but clearly it is.

Hervé Pagès (15:16:41) (in thread): > Hurry before my Spotify streaming dies again. It only works thanks to buffering at the moment…

Aaron Lun (15:17:28) (in thread): > Basically, the question is: what iscache_path, and why does it not exist?

Hervé Pagès (15:18:28) (in thread): > How do we answer this question?

Aaron Lun (15:18:53) (in thread): > Did you try thedebugthing I just posted?

Aaron Lun (15:19:05) (in thread): > Hold on, I can copy out the commands explicitly.

Hervé Pagès (15:19:24) (in thread): > oh I missed that post of yours. sorry

Hervé Pagès (15:20:06) (in thread): > The stupid Slack won’t mark there is a new answer if the answer is in a thread. TERRIBLE “feature”!

Hervé Pagès (15:20:47) (in thread): > but I’m going to wait the explicit commands since you offered

Aaron Lun (15:21:32) (in thread): > This would be the relevant part: > > path <- "zeisel-brain.Rmd" > prefix <- sub("\\.rmd$", "", path, ignore.case = TRUE) > cache_path <- file.path(paste0(prefix, "_cache"), "html") > cache_path <- paste0(cache_path, "/") > if (!file.exists(cache_path)) { > compileChapter(path) > } >

Aaron Lun (15:22:14) (in thread): > The implication is thatcache_pathdoes not exist if it callscompileChapter.

Hervé Pagès (15:23:48) (in thread): > > > library(rebook) > > path <- "zeisel-brain.Rmd" > > prefix <- sub("\\.rmd$", "", path, ignore.case = TRUE) > > cache_path <- file.path(paste0(prefix, "_cache"), "html") > > cache_path <- paste0(cache_path, "/") > > if (!file.exists(cache_path)) { > + compileChapter(path) > + } > # > # > # processing file: zeisel-brain.Rmd > # > # | > # | | 0% > # | > # |. | 2% > # ordinary text without R code > # > # > # | > # |... | 4% > # label: unref-setup (with options) > # List of 2 > # $ echo : logi FALSE > # $ results: chr "asis" > # > # > # | > # |.... | 6% > # inline R code fragments > # > # Quitting from lines 8-19 (zeisel-brain.Rmd) > # > # Error in Biocpkg("scRNAseq") : could not find function "Biocpkg" > > Error in compileChapter(path) : failed to compile 'zeisel-brain.Rmd' >

Aaron Lun (15:24:24) (in thread): > surreal. What is the value ofcache_path?

Aaron Lun (15:24:38) (in thread): > Should bezeisel-brain_cache/html/.

Hervé Pagès (15:24:52) (in thread): > > > cache_path > [1] "zeisel-brain_cache/html/" >

Aaron Lun (15:25:21) (in thread): > But… this exists, right? As per our checks above?

Hervé Pagès (15:26:06) (in thread): > Yes it exists

Aaron Lun (15:26:29) (in thread): > aw man. Windows. So despite the file existing,file.exists(cache_path)returnsFALSE.

Aaron Lun (15:26:52) (in thread): > Doesdir.exists(cache_path)do any better?

Hervé Pagès (15:27:05) (in thread): > > file.exists(cache_path) > [1] FALSE > > dir.exists(cache_path) > [1] TRUE

Hervé Pagès (15:27:17) (in thread): > Here you go

Hervé Pagès (15:27:35) (in thread): > Windows… again

Aaron Lun (15:27:46) (in thread): > Ugh. Reading?file.existsgives me: > > What constitutes a ‘file’ is > system-dependent, but should include directories. (However, > directory names must not include a trailing backslash or slash on > Windows.)

Aaron Lun (15:28:21) (in thread): > The problem is, if I don’t include the trailing backslash, the subsequent ****knitr**** calls fail because they don’t usefile.path()internally to assemble the path!

Aaron Lun (15:28:24) (in thread): > Ho ho ho.

Aaron Lun (15:28:40) (in thread): > Well. Okay, I’ll just switch todir.exists(). Thanks for the trouble.

Hervé Pagès (15:29:53) (in thread): > No problem. My Spotify streaming survived so we’re good:wink:(this thing is good at buffering)

Elana Fertig (21:51:31): > Thanks for catching that. Will check.

Melanie Loth (22:00:39) (in thread): > @Hervé PagèsDo you have a ballpark date for when the the move to riesling1 will be?

2020-10-20

Hervé Pagès (01:30:06) (in thread): > No clear roadmap yet. Could happen between a couple of weeks and a couple of months after the 3.12 release. It all depends on how well tokay2 will do with the 3.13 builds. tokay1 & tokay2 are old machines with an old OS, and the size of the project is getting close to the limit of what they can handle. So we’re going to have to start using much more powerful Windows builders like riesling1 very soon.

Jonathan Griffiths (03:23:16) (in thread): > Thanks for the information:slightly_smiling_face:

Jianhai Zhang (16:54:56): > Is the nightly build report delayed today?

Hervé Pagès (16:57:12): > There won’t be a new report today. We had a technical issue again. Sorry for the inconvenience.

Hervé Pagès (17:24:11): > @Jianhai ZhangToday’s builds have started already. riesling1 is a very fast builder and it should get to runR CMD buildon spatialHeatmap in the next couple hours or so (right now it’s reinstalling all software packages). I’ll let you know how it goes.

Jianhai Zhang (17:25:02): > Thanks for the updates.

Hervé Pagès (20:08:06): > Latest build on riesling1 still gives the following error: > > ... > Quitting from lines 327-328 (spatialHeatmap.Rmd) > Error: processing vignette 'spatialHeatmap.Rmd' failed with diagnostics: > invalid character in attribute value [9] > --- failed re-building 'spatialHeatmap.Rmd' > > This is on the latest spatialHeatmap (commit 5cb04388).

Jianhai Zhang (21:30:46): > On the settings athttps://bioconductor.org/checkResults/3.12/bioc-LATEST/Renviron.bioc, should I uncomment the 3 lines? > > #*R_CHECK_TIMINGS*="0" > #*R_S3_METHOD_LOOKUP_BASEENV_AFTER_GLOBALENV*=true > #*R_CLASS_MATRIX_ARRAY*=true >

2020-10-21

Hervé Pagès (02:34:39): > Renviron.biocis the file that iseffectivelyused on the build machines so if a setting is commented in the file it means it’s not used on the build machines. Note that these settings are unlikely to have anything to do with the ‘invalid character’ error. I suspect this is a locale/encoding problem. Here is what we have on the Windows builders: > > > Sys.getlocale(category = "LC_CTYPE") > [1] "English_United States.1252" > > What do you have on your Windows machine? You could try to set your LC_CTYPE to this value and see if you can reproduce the error.

Lori Shepherd (08:29:10) (in thread): > @Hervé Pagèscan you please announce on bioc-devel too when there will not be a build report so developers are aware. Many are looking for the new reports to check changes before release.

Hervé Pagès (12:47:28) (in thread): > will do

Jianhai Zhang (14:08:37): > It is the same.

Jianhai Zhang (14:10:21): > Can you provide these infor returned by sessionInfo()? R version, platform, runnning under, matrix products, locale?

Jianhai Zhang (14:11:39): - File (PNG): Screenshot from 2020-10-21 11-11-19.png

Jianhai Zhang (14:23:33): > andSys.getenv('R_GSCMD')?

Jianhai Zhang (14:53:25): > These are the values returned bySys.getenv()

Jianhai Zhang (14:53:37): - File (PNG): Screenshot from 2020-10-21 11-52-19.png

Jianhai Zhang (14:53:47): - File (PNG): Screenshot from 2020-10-21 11-52-50.png

Jianhai Zhang (17:43:17): > These are the sessionInfo() and Sys.getenv() in txt files.

Jianhai Zhang (17:43:40): - File (Plain Text): sessionInfo.txt - File (Plain Text): sys.getenv.txt

Hervé Pagès (20:15:24): > All right. After a painful debugging session on riesling1, here is what I found. > > The error occurs in the 1st call tospatial_hm()(“toyshm” code chunk in the vignette).spatial_hm()callsspatialHeatmap:::svg_df()internally, which itself callsxml2::read_xml()twice. The 2nd call toread_xml()(spatialHeatmap/R/svg_df.R, line 94) is the one that is failing. The input to this call is a temporary file previously generated bygrImport::PostScriptTrace(). On riesling1, this file isC:/Users/biocbuild/AppData/Local/Temp/3/RtmpGI7ksx/internal.ps.xmland starts with: > > <?xml version='1.0' encoding='ISO-8859-1' ?> > > <picture version='3' xmlns:rgml='[http://r-project.org/RGML](http://r-project.org/RGML)' source='C:Users^HiocbuildAppDataLocalTemp^CRtmpGI7k > sxinternal.ps' date='2020-10-21 17:51:39' creator='R (4.0.3)' > > > As you can see, the “source” attribute is broken (with invalid characters instead of/or backslahes\\). These characters breakxml2::read_xml(). > > Your code creates this XML file by callingPostScriptTrace()(spatialHeatmap/R/svg_df.R, line 93). The path for the input file looks like this: > > Browse[2]> file > [1] "C:\\Users\\biocbuild\\AppData\\Local\\Temp\\3\\RtmpGI7ksx\\internal.ps" > > It contains “escaped backslahes”, so is a valid path on Windows, at leastin theory, even though it tends to break code that is not handling these paths carefully. This seems to be the case ofgrImport::PostScriptTrace(), unfortunately. If you use forward slashes in the path, instead of backward slashes, then everything works as expected: > > > file <- "C:/Users/biocbuild/AppData/Local/Temp/3/RtmpGI7ksx/internal.ps" > > PostScriptTrace(file, "forward_slahes.ps.xml") > > cat(head(readLines('forward_slahes.ps.xml')), sep="\n") > <?xml version='1.0' encoding='ISO-8859-1' ?> > > <picture version='3' xmlns:rgml='[http://r-project.org/RGML](http://r-project.org/RGML)' source='C:/Users/biocbuild/AppData/Local/Temp/3/RtmpGI7ksx/internal.ps' date='2020-10-21 19:50:05' creator='R (4.0.3)' > > > <path type='fill' id='1'> > > > library(xml2) > > doc <- read_xml("forward_slahes.ps.xml") > > class(doc) > [1] "xml_document" "xml_node" > > I see that you’re using a combination oftempdir()andnormalizePath()to build the paths that you pass to the call toPostScriptTrace(). Unfortunately, these return paths containing “escaped backslashes”, even if the input path uses forward slashes in the case of the latter: > > > normalizePath("C:/Users/BIOCBU~1/AppData/Local") > [1] "C:\\Users\\biocbuild\\AppData\\Local" > > So the workaround would be to “fix” the input path toPostScriptTrace()right before calling the function. > > But IMOgrImport::PostScriptTrace()should really do something like this instead of producing broken XML files. Feel like reporting the issue to the grImport maintainers? > > Hope this helps. > > P.S.: I’m surprised that you were not able to reproduce the problem on Windows. After all it’s not an encoding/locale issue, just the typical/standard/universal/shared Windows pain, again.

Aaron Lun (20:35:56): > speaking of windows errors, beachmat is failing on tokay 32-bit but not on riesling 32-bit… I was just planning to let this slide, especially given that it was passing through fine previously and I didn’t change anything. Hopefully it’s just one of those sporadic failures.

Jianhai Zhang (20:41:05): > This is super helpful. Thanks for your time on the debugging! I am fixing it now.@Hervé Pagès

Jianhai Zhang (20:49:19): > And I will report this issue in grImport::PostScriptTrace() so that other users will not encounter this bug.

2020-10-22

Hervé Pagès (00:13:58) (in thread): > mmh.. probably not related to the error but how is it that filesbeachmat.Rcheck/tests_i386/testthat.Rout.failandbeachmat.Rcheck/tests_x64/testthat.Routbothcontain: > > * loading checks for arch 'i386' > ... > * loading checks for arch 'x64' > ... > > ?

Aaron Lun (03:34:22) (in thread): > hm, probably need to turn on--no-multiarchin the check-within-the-check.

Jianhai Zhang (15:40:02): > I am usingdevtools::check('spatialHeatmap', run_dont_test = F)on windows, no matterrun_dont_testisT or F, it is always checking examples with –run-donttest. Does anyone know the reason?

Jianhai Zhang (15:40:21): - File (PNG): Screenshot from 2020-10-22 12-37-34.png - File (PNG): Screenshot from 2020-10-22 12-37-12.png

Hervé Pagès (17:05:45): > Maybe you found a bug indevtools::check(), I don’t know, I don’t use devtools. It’s important to keep in mind thatdevtools::check()is just a nice (but apparently buggy) wrapper toR CMD check. In the mean time you can useR CMD check --run-donttestHopefully this one works as expected.

Jianhai Zhang (20:19:46): > HiHervé****, ****can you let me know the check results on commit 7c04c98 (spatialHeatmap) fromriesling1 if it is completed?@Hervé Pagès

Hervé Pagès (21:21:54): > I’ll let you know when it’s done.

Hervé Pagès (21:27:22): > I started to work on incremental builds every 4 hours. Every 4 hours we’ll build and check packages that have changed since the last run. Will be on nebbiolo1 (Linux) and riesling1 (Windows) only for now. The other builders don’t have enough power to handle the extra load. These builds will be informative only. No binary builds, no propagation. They should be ready in a couple of days.

Hervé Pagès (21:39:12): > @Jianhai ZhangBTW, what’s in the builds now is commit 2d71c84. That’s 3 commits before 7c04c98. Remember that the cut-off time for the daily builds is around 2:30 pm EST. These 3 commits all came after 5 pm EST.

Jianhai Zhang (21:45:00): > Ok. I thought the daily build time started at 6pm EST based on the message I received: Your package will be included in the next nigthly ‘devel’ build (check-out from git at about 6 pm Eastern; build completion around 2pm Eastern the next day)

Jianhai Zhang (21:48:07): > As long as it is the latest commit is used, that’s fine.

Hervé Pagès (21:48:14): > The times in that email are way off. We’ve shifted everything two hours earlier a few weeks ago. Before that shift the cut-off time was around 4:30 pm EST, not 6 pm EST. Please refer to this page for up-to-date information about the builds, including the cut-off time:https://bioconductor.org/developers/how-to/troubleshoot-build-report/

Jianhai Zhang (21:53:45): > Thanks for the clarification.

Jianhai Zhang (22:00:32): > What is the cut-off time for fixing errors/warnings tomorrow (Fri)?

Hervé Pagès (22:01:19): > 2:30 pm EST

Jianhai Zhang (22:11:07): > 2d71c84 will give the same error since the ‘\’ issue was addressed after it.

Hervé Pagès (22:29:34): > Yep, confirmed. I see the same error on riesling1.

Jianhai Zhang (22:34:57): > Then will the next build finish about 4 hours later?

2020-10-23

Hervé Pagès (01:07:00): > What builds? The daily builds? I’m not planning to change the daily builds. I only gave a heads-up that I’m working on incremental builds that will run every 4 hours. These would be inadditionto the daily builds. Provided as a convenience to reduce the time between a change to a package and its impact onR CMD build/R CMD check. Still not as convenient as on-demand builds, but closer. And not a substitute for a localR CMD build/R CMD checkby the developer. Also easier to manage and to keep under control from a build machine management point of view than on-demand builds. > But hey they’re not ready yet. This is work in progress. I said they should be ready in a couple of days.

Jianhai Zhang (01:22:16): > ok. so I can see the latest build report on Sat?

Hervé Pagès (02:15:42): > For the daily builds, there is normally a report every day. We’re going to keep the daily builds up and running even after tomorrow’s deadline (or maybe I should say today’s deadline because it’s already Friday for most people in the world). For spatialHeatmap, Saturday’s report will determine its inclusion in the BioC 3.12 release. Yes you’ll be able to see that report, of course, like any other daily report. > > I’m sorry to say but things are not looking too good at the moment. Your package got accepted after validation on all platforms by the Single Package Builder and a review. Then you decided to embark on last minute additions/changes. We discussed this and I gave you advice about that. In particular I warned you that in case things are not going as expected, you’d need to revert the changes. I also spent a lot of time giving you plenty of details about our builds, propagation, package versioning, etc… and even went on our Windows builder to debug the package for you. In retrospective, it is clear now that embarking on these last minute additions/changes was not such a good idea. Maybe it’s time to revert? If you decide to revert, don’t revert the version bumps: the version still needs to go up, even when you revert changes.

Charlotte Soneson (02:41:11) (in thread): > We got the same problem in yesterday’s build report forDuoClustering2018onriesling1. Just checking in to see that the advice to wait for the next build still stands and there isn’t anything we should try to fix before the deadline. > > > ### **** Examples > > > > sce_filteredExpr10_SimKumar4easy() > snapshotDate(): 2020-10-02 > see ?DuoClustering2018 and browseVignettes('DuoClustering2018') for documentation > downloading 1 resources > retrieving 1 resource > Warning: download failed > hub path: '[https://experimenthub.bioconductor.org/fetch/1537](https://experimenthub.bioconductor.org/fetch/1537)' > cache resource: 'EH1537 : 1537' > reason: database is locked > Error: failed to load resource > name: EH1537 > title: sce_filteredExpr10_SimKumar4easy > reason: 1 resources failed to download >

Jianhai Zhang (02:46:58): > Hi Hervé, I appreciate your time and effort on spatialHeatmap, especially the debugging. > > The current ‘\’ error was firstly reported on the commit 999fd84009f5da69db273e5a53431d9d328ee342 that passed all three builders without errors and warnings. In other words, the commit 999fd84 didn’t introduce errors during the review process but during the daily build. Then I was trying to reproduce the error, but failed all the time. Thanks to your debugging, I know it is a matter of ‘\’. I have fixed it in the newest commit, and on my windows 2019 server there was no error/warnings now. > > I was thinking about introducing new functions to this package, but without overcoming this error, I didn’t add the new functions.

Hervé Pagès (03:07:39): > ok, then we should be good

Jianhai Zhang (03:46:09): > I hope so. Thanks!

Jianhai Zhang (03:48:00): > The vignette was still the one generated on Oct 18.https://bioconductor.org/packages/3.12/bioc/vignettes/spatialHeatmap/inst/doc/spatialHeatmap.htmlDo you know why?

Jianhai Zhang (04:20:50): > Is the Bioconductor Docker image only available for linux builder, not windows?

Lori Shepherd (07:42:00) (in thread): > I’ll pop on Riesling later today to see if I can find any further information – we are also still working on the locking mechanism – sorry for the inconvenience

Hervé Pagès (09:07:28) (in thread): > This vignette is from version 0.99.16 of the package, which is the last version that propagated. Why would expect to see a more recent version of the vignette there?

Hervé Pagès (09:12:15) (in thread): > You should ask this on the#containerschannel.

Jianhai Zhang (12:55:15) (in thread): > Since the first figure was reduced in size and I nade changes.

Hervé Pagès (13:32:01) (in thread): > but did you bump the version after this change and did the new version of the package propagate?

Jianhai Zhang (13:35:27) (in thread): > The changes were in 0.99.16, no bumping.

Hervé Pagès (13:39:39) (in thread): > The vignette that is currently published is from the 0.99.16 version of the package (it’s even written at the beginning of the vignette). So either you made the change after the bump to 0.99.16 or you made it before but the change didn’t have the effect you anticipated. Did you build the vignette locally to check your changes?

Jianhai Zhang (13:45:11) (in thread): > Yes, the local change is expected.

Jianhai Zhang (13:45:57) (in thread): > The change was after the bump to 0.99.16

Jianhai Zhang (13:48:13) (in thread): > Let me bump to 0.99.18, Since 17 was used already.

Matt Stone (18:08:02): > Hi, I just pushed a small update (runtime improvement, no API changes) to devel a few minutes ago. I’m reading the thread between Jianhai and Hervé, and I was also under the impression that the cutoff time for the build was around 6pm EST based on the automated acceptance message after it passed review. Will there be any issues due to missing the 2:30pm EST cutoff today?

Martin Morgan (18:11:31): > The build times are more-or-less continuously adjusted, based on how long the builds take, what other tasks need to be performed, etc. Probably the best bet is to look at the ‘snapshot’ date on the most recent build (e.g.,http://bioconductor.org/checkResults/3.13/bioc-LATEST/at the top center), and bet that the next snapshot date will be at about the same time.

Martin Morgan (18:13:17): > I think there’s a location onbioconductor.orgwhere a more specific time is mentioned, but probably a pull request (onhttps://github.com/Bioconductor/bioconductor.org) could update that to say what I mention above – check the build report.

Matt Stone (18:16:55): > Thank you Martin. This is my first bioconductor package, so I’m a bit confused as to the distinction between today’s feature freeze and Tuesday’s code freeze. If the commit wasn’t included in today’s build, will there be any issues in the update being included in the 3.12 release?

Martin Morgan (18:28:00): > The ‘feature freeze’ is trying to guide you toward better practice – don’t introduce enormous changes at the last minute, as the most likely outcome is that the changes go awry. Instead, leading up to the release, focus on critical bug fixes and other more modest changes (e.g., improved documentation!) that will make the ‘release’ version of your package really shine. > > But so long as the changes you introduce now do not break the build of your package, they’ll be included in the 3.12 release.

Matt Stone (19:10:14): > Thanks Martin! That’s helpful to understand and I appreciate the advice

2020-10-24

Aaron Lun (20:25:37): > The wall is so green! It’s beautiful.

Hervé Pagès (20:38:25): > yeah we didn’t have such a good looking report for months:star-struck:

Hervé Pagès (20:38:47): > but we can still make it better, hopefully

Jianhai Zhang (22:54:33): > Hi Hervé, my package got a clear pass! Thank you for the supports!

2020-10-25

Hervé Pagès (01:01:47): > Just in time:wink:Thanks for fixing the package! There was a hiccup with propagation (the Windows binary didn’t propagate) but hopefully that will clear up tomorrow.

Jianhai Zhang (03:32:54): > Let’s see it tomorrow.

2020-10-26

Jianhai Zhang (12:48:44): > Is there anything I can do about existence in internal repo?

Jianhai Zhang (12:50:02): > screenshot

Jianhai Zhang (13:08:22): > I have a current error that islikely caused by Internet connection while downloading data.

Jianhai Zhang (13:17:05): > @Hervé Pagès

Jianhai Zhang (13:24:20): > http://bioconductor.org/checkResults/3.12/bioc-LATEST/spatialHeatmap/

Hervé Pagès (13:27:48): > not much I can do I’m afraid

Martin Morgan (13:28:16): > @Jianhai Zhangif the internet connection fails, then obviously there is nothing that the build system can do? It will try again the next night. Can you be specific about what you are concerned with? > > Since there has been no ‘version bump’ since the package was built and ‘pushed’ (tohttps://bioconductor.org/packages/3.12/bioc/html/spatialHeatmap.html) you can see that the current version of your package (0.99.8) is available across all platforms to users of Bioc-3.12. - Attachment (Bioconductor): spatialHeatmap (development version) > The spatialHeatmap package provides functionalities for visualizing cell-, tissue- and organ-specific data of biological assays by coloring the corresponding spatial features defined in anatomical images according to a numeric color key.

Jianhai Zhang (13:30:20): > my concern is if the error affects the release, since today the code will be frozoned.

Lori Shepherd (13:31:04): > the code won’t be frozen today – the current version will become the release – so to debug you would have to update on two branches instead of one

Hervé Pagès (13:31:55): > right, even though in that case there’s not much to debug

Nicholas Cooley (15:54:39): > I have a funny question about versions and devel and release, i’m not sure if this is just naive but; when i make changes to my package locally, and then push those to both my github, and the bioconductor remote, i bump my version by x.y.z + 1, that “z” id is only seen by devel? So when devel rolls over to release i just pull the updated description from upstream to local and i’m ready to work again? and version numbers do, or don’t bump no matter whether i’ve made changes or not?

Marcel Ramos Pérez (16:14:49): > when you push, you’re usually pushing toupstream/masterin yourgit@git.bioconductor.org:packages/MyPackageremote (usually namedupstream). So if you have0.99.1in the devel when the release comes around a new branch is createdRELEASE_3_12in theupstreamremote and the version is bumped to1.0.0(yis even in release). You then dogit fetchandgit mergeor simplygit pullfrom theupstream. You then push both the version bump for release and devel to your GitHub location

Nicholas Cooley (16:36:09): > ok so isupstreamequivalent to or always devel ?

Marcel Ramos Pérez (16:37:08): > upstreamis a remote which can have branches (masterandRELEASE_X_XX) depending on how long your package has been in Bioc

Marcel Ramos Pérez (16:39:04): > you can check them usinggit branch -a

Alan O’C (16:44:13): > You can list remotes withgit remoteand show details for specific remotes withgit remote show originwhereoriginis the name of the remote

Nicholas Cooley (16:44:56): > ok, so if i’ve got1.0.0in the current release andgit fetch+git mergeget me1.1.0, when i make all my edits, bump the version to1.1.1, and the new build passesR CMD BiocCheck, thengit push upstream masterputs the new version where?

Marcel Ramos Pérez (16:45:00): > Thanks Alan. alsogit remote -vgives you locations

Marcel Ramos Pérez (16:46:00): > ifupstreamisgit@git.bioconductor.org:packages/MyPackagethen the new version is sent to Bioconductor

Marcel Ramos Pérez (16:46:22): > but you have to be on themasterbranch

Marcel Ramos Pérez (16:46:59) (in thread): > If you do fetch and merge on the release branch you will get1.0.0not1.1.0

Nicholas Cooley (16:47:38): > and if my branch was set previously withgit checkout master, so i would still be, supposedly? Sorry, i’m just trying to make sure i’ve done this correctly!

Nicholas Cooley (16:48:41): > If that’s right, and my update passes it’s builds, it will get renamed to1.2.0on the next release?

Marcel Ramos Pérez (16:49:19): > yes if pre-release the master branch was1.1.x

Marcel Ramos Pérez (16:50:15): > Feel free to create an issue or PR for the git workflow outlined here if it’s not clear or missing information. Feedback appreciatedGitHub Repo:https://github.com/bioconductor/bioconductor.org

Hervé Pagès (16:55:28): > I don’t know where this is going but I want to emphasize the fact that we will create the RELEASE_3_12 branch and bump the versions in the new branch and in master the day before the release. So there’s nothing to do on your side as far as branch creation and version bumps are concerned. From your perspective, the rule to bumpzinx.y.zwhen you make changes to your package always applies, whatever branch you are on, and whether this is before or after the release.

Marcel Ramos Pérez (16:58:39) (in thread): > Thanks Hervé! Also, developers should remember topullbeforepushing commits (to avoid merge conflicts) immediately after the release

Nicholas Cooley (16:59:37): > The github workflow is fairly helpful, it’d be nicer it if was a little more explicit, and wasn’t split up across multiple pages. Like i just want tomake surethat i’m typically only ever bumpingzinx.y.zand that when i push to upstream, it’s specifying what at least i’ve been thinking of as “devel”.

Hervé Pagès (17:00:42) (in thread): > Right. As a general rule, it’s always a good idea to pull before committing/pushing, even in normal times.

Marcel Ramos Pérez (17:08:20) (in thread): > yes, as Hervé said.. Theupstreamremote is not necessarilydevelper se. It is only when you’re on themasterbranch. Bug fixes can happen on theRELEASE_X_XXbranch and those get pushed toupstream/RELEASE_X_XX. Seehttp://bioconductor.org/developers/how-to/git/bug-fix-in-release-and-devel/

Hervé Pagès (23:57:35) (in thread): > @Nicholas CooleyYou’re welcome to bring suggestions for improving the git workflows to the#bioc_gitchannel.

2020-10-27

Jianhai Zhang (13:00:17): > The same Internet connection error occurred on other different builders. Does it affect the release?https://master.bioconductor.org/checkResults/3.12/bioc-LATEST/spatialHeatmap/

Lori Shepherd (13:08:50): > it will not effect the package moving forward to release – however – you should try to get to the route of the connectivity issue – if your internet resource is that unstable than it may not be useful – also for downloading files from the web we highly recommend a caching mechanism so that if a file become unavailable the code does not break (see BiocFileCache)

Jianhai Zhang (13:13:52): > Thanks for the advice.

2020-10-28

Lori Shepherd (15:02:58): > Bioconductor 3.12 is Released! Thank you to all developers and community members for contributing to the project! Please see the full release announcement:Bioc 3.12

Aaron Lun (16:53:07): > Unrelated, but I guess we’ll see it again on Friday: looks like malbec’s pandoc isn’t working properly, > > # /usr/bin/pandoc +RTS -K512m -RTS pancreas.utf8.md --to html4 --from markdown+autolink_bare_uris+tex_math_single_backslash --output pancreas.html --lua-filter /home/biocbuild/bbs-3.12-bioc/R/library/bookdown/rmarkdown/lua/custom-environment.lua --lua-filter /home/biocbuild/bbs-3.12-bioc/R/library/rmarkdown/rmarkdown/lua/pagebreak.lua --lua-filter /home/biocbuild/bbs-3.12-bioc/R/library/rmarkdown/rmarkdown/lua/latex-div.lua --metadata-file /tmp/Rtmp7VaSZ6/file3f6833949469 --email-obfuscation none --wrap preserve --standalone --section-divs --table-of-contents --toc-depth 3 --template /home/biocbuild/bbs-3.12-bioc/R/library/bookdown/templates/gitbook.html --highlight-style pygments --number-sections --css style.css --include-in-header /tmp/Rtmp7VaSZ6/rmarkdown-str3f68652a9be6.html --mathjax --filter /usr/bin/pandoc-citeproc > # Unknown option --metadata-file. > # Try pandoc --help for more information. > # Error : pandoc document conversion failed with error 2 > > Occurs for both OSCA and SingleRbook.

Lori Shepherd (17:43:37): > Thanks we will investigate

Hervé Pagès (17:58:39) (in thread): > Maybe you are calling the old pandoc that is part of Ubuntu 18.04? Because it’s too old we’ve installed a more recent pandoc on the malbecs. It’s in/usr/local/binand it’s in the path: > > biocbuild@malbec1:~$ /usr/bin/pandoc --version > pandoc 2.1 > Compiled with pandoc-types 1.17.3, texmath 0.10.1, skylighting 0.5.1 > Default user data directory: /home/biocbuild/.pandoc > Copyright (C) 2006-2018 John MacFarlane > Web:[http://pandoc.org](http://pandoc.org)This is free software; see the source for copying conditions. > There is no warranty, not even for merchantability or fitness > for a particular purpose. > > biocbuild@malbec1:~$ /usr/bin/pandoc --help | grep metadata-file > > biocbuild@malbec1:~$ which pandoc > /usr/local/bin/pandoc > > biocbuild@malbec1:~$ pandoc --version > pandoc 2.7.3 > Compiled with pandoc-types 1.17.5.4, texmath 0.11.2.2, skylighting 0.8.1 > Default user data directory: /home/biocbuild/.local/share/pandoc or /home/biocbuild/.pandoc > Copyright (C) 2006-2019 John MacFarlane > Web:[http://pandoc.org](http://pandoc.org)This is free software; see the source for copying conditions. > There is no warranty, not even for merchantability or fitness > for a particular purpose. > > biocbuild@malbec1:~$ pandoc --help | grep metadata-file > --metadata-file=FILE >

Aaron Lun (17:59:18) (in thread): > I don’t personally call any pandoc myself, it’s all done viarmarkdown.

Hervé Pagès (18:00:22) (in thread): > Damn, then I would say that it’s rmarkdown that is not functioning properly.

Hervé Pagès (18:02:39) (in thread): > Why we only start seeing this error now? Did you start using pandoc’s metadata-file option recently? Or maybe it’s rmarkdown that started to use this option recently?

Hervé Pagès (18:03:10) (in thread): > oh oh:https://cran.r-project.org/web/packages/rmarkdown/index.htmlPublished on Oct 21! - Attachment (cran.r-project.org): rmarkdown: Dynamic Documents for R > Convert R Markdown documents into a variety of formats.

Hervé Pagès (18:04:15) (in thread): > it’s been a while since we didn’t have a new round of knitr/rmarkdown/pandoc breaking changes:face_with_rolling_eyes:

Hervé Pagès (18:10:36) (in thread): > I removed the Ubuntu pandoc on malbec1: > > hpages@malbec1:~$ dpkg-query -s pandoc > dpkg-query: package 'pandoc' is not installed and no information is available > Use dpkg --info (= dpkg-deb --info) to examine archive files, > and dpkg --contents (= dpkg-deb --contents) to list their contents. > > Let’s see if this helps. > But rmarkdown should really call whatever pandoc is in the path.

Aaron Lun (18:22:21) (in thread): > ¯*(ツ)*/¯

Aaron Lun (18:22:27) (in thread): > I dunno, I just press the button

Hervé Pagès (18:27:25) (in thread): > well now you know, so you can do something about it

Aaron Lun (18:33:02) (in thread): > I mean, can I? I don’t see anyway to tell rmarkdown to use a different pandoc.

Hervé Pagès (18:35:06) (in thread): > no but you can report the problem so it gets fixed upstream

Hervé Pagès (18:35:41) (in thread): > I’m not saying you should do it, but it’s always an option.

2020-10-29

Hervé Pagès (12:55:29) (in thread): > Wait, it looks like we have/usr/binbefore/usr/local/binin the PATH for crontab jobs. Arghh… Changing this now. Ok so there’s nothing wrong with rmarkdown and nothing to report. Sorry for the not-so-useful suggestion.

Jianhai Zhang (22:34:38) (in thread): > RegardingYou then push both the version bump for release and devel to your GitHub location, I am confused. > > If I push new commits to both RELEASE_3_12 and master (devel) on upstream, will the package installed byBiocManager::install()include the new commits pushed to RELEASE_3_12? If so, users can get different versions of the package if they installed at different times.@Marcel Ramos Pérez

2020-10-30

Aaron Lun (02:54:45): > I think I might integrate the csawUserGuide and chipseqDB as a book, which should make it easier to read.

Hervé Pagès (03:04:27): - File (JPEG): the_more_the_merrier.jpg

Marcel Ramos Pérez (09:14:50) (in thread): > It depends on your Bioconductor version either3.12release or3.13devel. See the BiocManager vignette herehttps://cran.r-project.org/web/packages/BiocManager/vignettes/BiocManager.html

Marcel Ramos Pérez (09:15:52) (in thread): > Users get the version of the package that corresponds to the Bioconductor version installed

Martin Morgan (13:12:27): > Some additional work needs to be done to integrate the books into the Bioconductor web site – I don’t think there is any link, for instance, to the SingleR book (well, ok, from a SingleR package vignette…)

Lori Shepherd (13:21:14): > maybe@Hervé Pagèsand I can coordinate next week about this – It seems like it would be nice to have a link in the biocViews page, Just need to figure out what would have to be present besides a “books” biocView for it to be possible –

Mike Smith (13:31:27): > Maybe this is the wrong channel, but do people think of books as philosophically different from the workflows? I know from a technical point of view it’s different as there’s multiple Rmd files and the rendering is more complex than a standard vignette, but content wise they both seem to fit in the space for topic-focused documentation that spans multiple packages. Maybe the workflows are more “task driven” and a book is “for reference”, but think that’s probably me trying to think about the definitions of the words rather than the reality of the content. I wonder if, for example, theRNA-seq workflowwas written today, it would actually be a book now?

Aaron Lun (13:32:23): > as soon as you have multiple dependent Rmarkdowns, they’re basically a book.

Aaron Lun (13:32:59): > I would say that, if you ever find yourself saying, “and here we’ll use the object from the previous vignette”, you’re really writing a book.

FelixErnst (13:33:50): > I like the distinction between task driven and reference

Hervé Pagès (14:53:32): > Oops, the book builds failed to grab the RELEASE_3_12 branchGlad I caught this in time before propagation. I fixed the problem and will run these builds again as soon as there’s a good window for it.

Jianhai Zhang (15:13:18) (in thread): > I read this vignette. Let’s say users use Bioc 3.12. After I pushed commits to RELEASE_3_12, will users get the commits by using Bioc 3.12? In other words, is the release version fixed till next release? Or it can be updated before next release?

Marcel Ramos Pérez (15:14:15) (in thread): > Generally, yes after changes are propagated by the BBS. The release version is generally unchanged unless there is a bug fix to be made..

Marcel Ramos Pérez (15:15:00) (in thread): > We’d like it to be a stable release. Continual development happens on Bioc devel

Jianhai Zhang (15:16:48) (in thread): > Then new functionalities should only be pushed to devel, not relase. Is it right?

Marcel Ramos Pérez (15:43:27) (in thread): > Yes

Marcel Ramos Pérez (15:45:57) (in thread): > see under “Scenarios for code update:”http://bioconductor.org/developers/how-to/git/

Jianhai Zhang (15:46:39) (in thread): > Thanks!

Jianhai Zhang (16:36:32) (in thread): > I bumped the version to 1.1.1 in the deve branch, and theWARNING: y of x.y.z version should be even in releasearose in BiocCheck. Will this error disappear on Bioc daily check on devel?

Marcel Ramos Pérez (18:01:41) (in thread): > perhaps you’re running the release version ofBiocCheck? SeeBiocManger::version()andBiocManager::valid()

Jianhai Zhang (18:05:54) (in thread): > Yes, it is release version.

Jianhai Zhang (18:15:51) (in thread): > But why is there no warning on version 0_99_*?

Jianhai Zhang (18:16:32) (in thread): > Same release verion of BiocCheck.

Mike Smith (18:50:02) (in thread): > I guess the my point was mostly thinking about their integration into the BioC website. Are Workflows and Books be distinct from a user perspective, or are they two approaches to meet the same need. If it’s the later then maybe they should be presented to users together, rather than separate categories of content. Maybe a book has such a large volume of content compared to the workflows that it should be classed separately? But that RNA-seq document is pretty long, and I could imagine considering splitting into chapters if given the opportunity at the time. I’m not particularly convinced one way or the other, just thought it would be interesting to gauge people’s opinions before someone undertakes work on the website.

Hervé Pagès (20:06:07): > Yes, these are good questions and getting some feedback on this would help us take the right approach. One thing is that, at the moment, unlike what we do for workflows, we don’t propagate the source tarballs associated with the books. So they don’t end up in a package repo and you can’t install them viaBiocManager::install(). We could change that. Would this have a lot of value? I’m not sure. But it would be easy to do. We just need to know what people think.

2020-10-31

Vince Carey (10:50:02): > One of the questions I am facing with respect to books is change management. While a workflow can be pretty reasonably frozen to a software release, it seems less appropriate to freeze a book, which I think is intended to have broader access and to be very accurate with respect to current state of the art. One really wants to “pulp” an out-of-date book – IMHO. Maybe we just have a string of editions and clear pointers to readers that there is a more recent edition when in fact that is so. I don’t know if this discipline is already in place, but it seems salient for books specifically.

Hervé Pagès (17:35:39): > Just to clarify right now the books that are in the build system are tied to a given version of Bioconductor. For example the release version of the SingleRBook is deployed herehttps://bioconductor.org/books/3.12/SingleRBook/and its devel version will be deployed herehttps://bioconductor.org/books/3.13/SingleRBook/. Only the most recent version of the 3.12 or 3.13 book is deployed and accessible. In that aspect it’s no different from how we manage workflows or software packages. > To make books visible on the website, as a starter we could add a link to the Documentation box herehttps://bioconductor.org/help/to a page that lists the books that are in the current release, like what we do already for workflows. The index of workflows is herehttps://bioconductor.org/packages/release/workflows/but for books it would be herehttps://bioconductor.org/books/release/and it could look a little bit different e.g. we don’t need to stick to the table presentation, would list all the authors, maybe display the DESCRIPTION in addition to the Title, the licence, etc… - Attachment (bioconductor.org): Assigning cell types with SingleR > The SingleR book. Because sometimes, a vignette just isn’t enough.

Hervé Pagès (20:40:24): > OK I’ve put something there:https://bioconductor.org/books/3.12/It’s a static page for now but it will need to be dynamically generated with ORCIDs, DOIs, etc… It’s linked from there:https://bioconductor.org/help/

2020-11-01

Martin Morgan (09:47:34) (in thread): > I think that these books should be treated like regular packages, added tobioconductor.org/packages, with a standard landing page, and installable viaBiocManager::install(). I think a pretty reasonable, and even exciting, use case is that the Rmd files in the package could be used as templates for individuals performing their own analyses – these are excellent resources, and the chapters are, more or less, stand-alone and very amenable to a few brief edits to ‘swap out’ the example data set and use one’s own data. > > The workflows page is the same as the software (‘bioc’) pagehttps://bioconductor.org/packages/3.12/bioc/and there is value in consistency.

Hervé Pagès (15:23:58) (in thread): > So if nobody objects I’ll create thebioconductor.org/packages/X.Y/booksrepos and propagate the source tarballs there. Someone will need to take care of adding these repos toBiocManager::repositories(). Not entirely clear that we should use exactly the same template for the book landing pages as for the workflows. Maybe some tweaking will be needed.

Hervé Pagès (23:45:28): > New report for 3.12 books:https://bioconductor.org/checkResults/3.12/books-LATEST/

2020-11-02

Aaron Lun (00:06:43): > Well, at least we’re getting further through it.

Aaron Lun (00:16:53): > Before I get into that, we’re in another sort of pickle. It seems like the following (on 3.13) no longer works to construct DataFrames with S4 objects as columns: > > out <- DataFrame(X=I(DataFrame(Y=1))) > ## Warning message: > ## In class(x) <- unique.default(c("AsIs", oldClass(x))) : > ## Setting class(x) to multiple strings ("AsIs", "DFrame", ...); result will no longer be an S4 object > > Pulling outout$Xgives me: > > <S4 Type Object> > attr(,"rownames") > `\001NULL\001` > attr(,"nrows") > [1] 1 > attr(,"listData") > attr(,"listData")$Y > [1] 1 > > attr(,"elementType") > [1] "ANY" > attr(,"elementMetadata") > `\001NULL\001` > attr(,"metadata") > list() > attr(,"class") > [1] "DFrame" >

Aaron Lun (00:16:56): > Uhgh. ThisI()strategy is used a lot by SCE and related packages. Any idea for a fix?

Aaron Lun (00:32:49) (in thread): > Hm. Can’t repro the OSCA failure.

Aaron Lun (00:36:26) (in thread): > I wonder if you could do me a favor: > 1. Login to nebibblio > 2. Clone the OSCA repo and go tovignettes/book. > 3. Comment out lines 161-163 and lines 183-185 inmerged-hsc.Rmd. > 4. Runrmarkdown::render("merged-hsc.Rmd")and send me the HTML file.

Aaron Lun (00:37:06) (in thread): > The error itself is a fail-safe to ensure that the plain-English text of the chapter isn’t off-the-rails relative to the results that are being shown. The problem is that the results are not consistent between nebiblio and my laptop. Which is already somewhat concerning.

Aaron Lun (00:42:16): > It seems like the above use ofI()was a bit of a hack in the first place; it would explain why my inner DF’s always lost theirpackageattribute in theclass(). Seems like we could do with something like anS4Vectors::asis()to protect objects going intoDataFrame().

Hervé Pagès (01:14:34): > https://stat.ethz.ch/pipermail/r-devel/2020-October/080038.htmlLet’s wait and see what R-core decide to do about this but yeah we might need to go for a proprietary solution.

Hervé Pagès (01:15:25) (in thread): > going to nebbiolo1 now…

Aaron Lun (01:19:16): > one step ahead, I see

Aaron Lun (01:19:48): > I wonder whether you could makeIan implicit generic.

Hervé Pagès (01:29:20) (in thread): > rmarkdown::render("merged-hsc.Rmd")still running after 10 min…

Aaron Lun (01:30:08) (in thread): > Sounds about right, it compiles three other reports at the same time.

Aaron Lun (01:30:28) (in thread): > Ordinarily those would be compiled before we got to that chapter, so theextractCached()calls would just be fetching content from theknitrcache.

Hervé Pagès (01:33:18): > Problem is we’d need to be able to dispatch on a method for S4 objects. I don’t see a simple way to do that. We could also overrideI()with a plain function that usesisS4()to do its own dispatch.

Hervé Pagès (01:35:32) (in thread): > Only at 27%. Looks like I’ve plenty of time to take the trash out.

Hervé Pagès (01:57:12) (in thread): > Done. Just sent the file to you.

Aaron Lun (02:23:59) (in thread): > thanks.

Aaron Lun (02:24:09) (in thread): > oh bum. The images aren’t embedded

Aaron Lun (02:35:04) (in thread): > But there’s enough debugging nfo here, so shouldn’t matter

Aaron Lun (02:47:26) (in thread): > Hm. Disturbing.

Aaron Lun (02:56:59) (in thread): > fxed.

Aaron Lun (23:49:21): > The conversion of csawUserGuide to a book has begun.

Aaron Lun (23:49:48): > This will also likely absorb chipseqDB’s content as well, so that package will turn into a stub.

2020-11-03

Aaron Lun (00:56:51): > finally! after several years, I have converted this tikz image of directional read extension into an R plot equivalent.

Aaron Lun (00:57:05): > Believe it or not, this was the blocker for why the user’s guide was not converted to Rmarkdown earlier.

Alan O’C (07:26:04): > Can you not use tikz in Rmds if it’s compiled to PDF? I have a feeling I managed that before

Aaron Lun (10:57:44): > no, these are going to bookdown.

Aaron Lun (10:57:48): > well, bookdown to HTML.

Hervé Pagès (15:22:09): > Looks good

Aaron Lun (15:29:57): > ATLAST

Aaron Lun (15:30:03): > hooray

Aaron Lun (15:30:09): > one month after our last success

FelixErnst (15:30:28): > how long before that?

FelixErnst (15:31:20): > Just asking for extrapolation:grin:

Aaron Lun (15:39:36): > months.

Aaron Lun (15:39:40): > maybe 2-3 months.

Aaron Lun (15:39:49): > In those days, I was manually building it on my laptop.

Aaron Lun (15:40:16): > Hm. 3.6 hours. I wonder if I added more stuff.

FelixErnst (15:40:36): > Keeping my finders crossed:slightly_smiling_face:

2020-11-04

Aaron Lun (23:32:07): > my god. I have so many defunct functions, i have to section my defunct page.

Aaron Lun (23:39:47): > oh. 2k deletions. It’s beautiful.

2020-11-05

Aaron Lun (02:38:19): > Major performance regressions from ****testthat****. Tests are taking about 50% more time.

FelixErnst (02:39:38): > why and how? Please spill the details. every expect_that block or every test_that block?

Aaron Lun (02:39:48): > I’m trying to debug it now

FelixErnst (02:40:17): > Do you have access to 4.0.2?

Aaron Lun (02:40:23): > Seems that individually running each file shows no difference with old vs new, buttest_checkis much slower.

FelixErnst (02:41:42): > I had very weird case of slow running code, when I updated to 4.0.3. I am not saying that this was Rs fault, because that peace of code was a dumpster fire anyway, but before it worked fine and after it didn’t

FelixErnst (02:42:08): > Fixed it and it ran with same time again

Aaron Lun (02:44:34): > If I had to guess - I would say that it is not interacting well with the forking.

FelixErnst (02:47:58): > in your code or testthat?

Aaron Lun (02:49:01): > the forking is in my code, but I dunno.

Aaron Lun (02:50:24): > switching all MulticoreParam instances to SerialParam gives me some better timings, but also throws in a whole lot of random errors, so… who knows what’s going on.

Aaron Lun (03:12:51): > I’m pretty confident that testthat is not interacting nicely with SnowParam.

FelixErnst (03:20:41): > Does that mean it is a Windows specific behaviour? SnowParam is the only option there, isn’t it?

Aaron Lun (03:20:55): > No, I’m on linux.

FelixErnst (03:21:03): > I guessed so

FelixErnst (03:22:44): > I am not sure about the details, but SnowParam is starting a new R session isn’t it, where MulticoreParam is basically using fork. Is that correct?

Aaron Lun (03:27:14): > Yup.

Aaron Lun (03:27:48): > The SnowParam penalty is brutal. Without it, tests take 270 seconds. With it, tests take 440 seconds. It’s insane.

FelixErnst (03:28:38): > how many workers?

Aaron Lun (03:28:46): > 2.

FelixErnst (03:31:52): > What happens with 4?

FelixErnst (03:32:05): > Does it increase again?

Aaron Lun (03:32:31): > ¯*(ツ)*/¯

Aaron Lun (03:33:12): > testthat added its own parallel support. I suspect the two are not playing nice.

FelixErnst (03:35:22): > That would have been my next stop. I am just asking, because depending on runtime of single workload vs. time it takes to load the session, I wouldn’t have been too shocked by the number, if you hadn’t said it was worse than before.

Aaron Lun (03:36:20): > the problem is, sourceing each test file does not lead to the slowdown.

Aaron Lun (03:36:40): > so it’s something particular with howtest_checksets up its environment.

FelixErnst (03:38:04): > MulticoreParam doesn’t suffer the same penalty?

Aaron Lun (03:55:57): > no.

Aaron Lun (03:58:59): > 20 seconds extra per SnowParam hit, basically.

Aaron Lun (03:59:07): > god knows what it’s doing in there.

FelixErnst (03:59:32): > Have you readhttps://www.tidyverse.org/blog/2020/10/testthat-3-0-0/? - Attachment (tidyverse.org): testthat 3.0.0 > testhat 3.0.0 brings a raft of major improvements including snapshot testing and parallel testing. It also includes a new “edition” that allows you opt-in to a set of substantial improvements that are not backward compatible.

Aaron Lun (04:00:11): > yes

Aaron Lun (04:00:32): > I stared at it for a long time to try to figure out what happened.

Aaron Lun (04:09:07): > here is my last insight before I go to bed: > > test_dir("testthat", package="scran") # 48 seconds > > Versus: > > test_dir("testthat") # 30 seconds. >

Aaron Lun (04:13:14): > Some naughty shenanigans with environment cloning, it seems.

FelixErnst (04:14:18): > So basically, what everybody is told not do

FelixErnst (04:14:41): > Who would have guessed

Aaron Lun (04:14:56): > And indeed, you can make the tests~great~fast again with: > > test_check("scran", env=.GlobalEnv) >

Aaron Lun (04:15:16): > Basically forcing it to not do naughty things.

Aaron Lun (04:15:19): > I’m going to sleep.

FelixErnst (04:15:46): > thanks for the walkthrough and noticing the issue

Hervé Pagès (12:36:16): > Ok so maybe testthat 3.0.0 is the reason why the builds slowed down in the last couple of days and are now taking about 1.5 hour longer on all platforms. This could also explain why we are seeing a higher number of TIMEOUTs as usual on Windows. Ouch! That’s a serious bummer because our old Windows machines (tokay1 & tokay2) are already at the limit of what they can handle and our new super duper Windows builder (riesling1) died on release day.:worried:

FelixErnst (12:57:26): > Is there any hope, that letting them know about the slow down might make them reconsider some changes?

Aaron Lun (13:00:58): > It’s probably unintended, but a MWE is tricky.

Hervé Pagès (13:03:11): > Will look into this. I’m trying to figure out if that claim is actually true, how it can easily be demonstrated, etc… If parallel evaluation is the default maybe it can be changed via some setting.

Aaron Lun (13:04:01): > testthat’s parallel mode isn’t the default, so that isn’t directly the problem.

FelixErnst (13:04:08): > Apparently it is not the default, if I understand their blog entry

Aaron Lun (13:24:18): > FYI, if you were trying to reproduce my numbers above, I deleted everytest-*file except fortest-pairwise-t.R.

Hervé Pagès (15:26:16): > ~788~770 software packages usetestthatin BioC 3.12. > > Here are the timings (proc.time’s displayed bytestthat) on tokay1 (Windows) for 20 packages selected (almost) randomly, before and after the update fromtestthat2.3.2 to 3.0.0: > > ----- i386 ---- ----- x64 ----- > version before after before after > AllelicImbalance 1.28.0 20.84 63.71 31.92 66.62 > basilisk 1.2.0 NA 717.82 NA 1016.89 > beachmat 2.6.0 399.70 819.65 245.78 255.62 > BiocFileCache 1.14.0 48.90 53.20 42.64 55.89 > BiocSingular 1.6.0 59.26 62.54 61.84 59.20 > bsseq 1.26.0 357.04 1413.62 385.04 NA > DelayedMatrixStats 1.12.0 994.31 1059.82 977.29 960.76 > DESeq2 1.30.0 264.89 335.20 312.71 348.40 > ensembldb 2.14.0 445.39 401.29 292.98 265.43 > mbkmeans 1.6.0 319.50 264.25 294.00 274.32 > Modstrings 1.6.0 67.04 44.76 70.93 60.10 > SingleCellExperiment 1.12.0 28.15 39.85 29.64 42.70 > Structstrings 1.6.0 97.85 109.87 117.48 136.31 > TreeSummarizedExperiment 1.6.0 28.14 22.89 23.07 21.32 > transite 1.8.0 220.87 209.56 230.71 244.39 > tRNA 1.8.0 45.59 38.56 39.73 49.95 > tximport 1.18.0 242.21 217.57 231.71 180.04 > VariantExperiment 1.4.0 25.54 19.93 19.36 22.34 > xcms 3.12.0 388.43 1795.46 377.57 NA > zinbwave 1.12.0 289.23 239.04 274.15 245.39 > ------------------------------------------------------------------------ > total 4342.88 7210.77 3295.94 3288.78 > > (basilisk excluded from i386 & x64 totals, bsseq and xcms excluded from > x64 totals) > > Theproc.timedifference varies greatly across packages (some packages don’t seem affected at all) but overall there’s a clear slowdown for the i386 (32-bit) tests but not for the x64 (64-bit) tests. Weird! > > Although the “before” and “after” timings correspond to build runs that happened 5 days apart (Oct 31 and Nov 5) so other changes could play in e.g. changes that happened in other dependencies.

Alan O’C (15:30:24): > That’s odd, I assume Aaron’s not stubborn enough to stick with i386 until now so there may be multiple sources of slowdown

Hervé Pagès (20:43:36): > Can’t believe knitcitations got removed from CRAN:http://cran.r-project.org/package=knitcitationsThis breaks at least 20 packages in release and devel! What’s next, knitr?

Marcel Ramos Pérez (20:56:38): > It looks like a dependency violated CRAN policyhttps://cran.r-project.org/web/packages/RefManageR/index.html

Hervé Pagès (21:32:06): > Looks like they couldn’t make their mind: > > License: GPL-2 | GPL-3 | BSD_3_clause + file LICENSE > > The exact meaning of the pipe must be left to the interpretation of the user… or the lawyers:grin:

2020-11-06

Aaron Lun (00:41:12): > Further boiling of the testthat problem suggests it’s something in BiocGenerics’ environment.

Aaron Lun (00:55:07): > actually, reasonably confident that it affects all S4-containing packages.

Aaron Lun (01:35:03): > OMG. I think I have it.

Aaron Lun (02:10:31): > :sob:that’s 4 hours of my life I wno’t get back.

Aaron Lun (02:11:07): > issue incoming in BiocParallel.

Hervé Pagès (02:11:16): > but you’ll get my immense gratitude in return

Hervé Pagès (02:13:14): > BTW, I’ve just committed an ugly workaround for R 4.1’s misbehavedI():https://github.com/Bioconductor/S4Vectors/commit/eed02abf8ab1a600351890028e1a89aa9176a76e

Aaron Lun (02:45:22): > It is done:https://github.com/Bioconductor/BiocParallel/issues/127

Hervé Pagès (02:53:16): > Excellent. Just out of curiosity I’ve rolled back testthat from 3.0.0 to 2.3.2 on malbec1 for the current builds so I can do more comparisons tomorrow. > That being said… good night

Martin Morgan (03:01:04) (in thread): > Thanks for the detective work Aaron.

Aaron Lun (03:23:09): > There is still a considerable slow-down, but I suspect that’s more because testthat is a lot more pedantic about tracking warnings, and it’s spending a lot of overhead on warnings that I previously just ignored.

Hervé Pagès (18:54:03): > But your fix should address the overall slowdown I observed with the builds. Here are the timings on malbec1 (Linux) for the same set of packages as yesterday, before and after the update fromtestthat2.3.2 to 3.0.0 (but without your fix yet): > > version before after > AllelicImbalance 1.28.0 24.12 23.73 > basilisk 1.2.0 401.90 414.14 > beachmat 2.6.0 226.90 283.46 > BiocFileCache 1.14.0 59.77 57.12 > BiocSingular 1.6.0 145.27 128.70 > bsseq 1.26.0 112.25 150.31 > DelayedMatrixStats 1.12.0 1077.70 1031.43 > DESeq2 1.30.0 323.67 322.42 > ensembldb 2.14.0 276.94 236.00 > mbkmeans 1.6.0 175.53 180.75 > Modstrings 1.6.0 56.96 48.62 > SingleCellExperiment 1.12.0 36.38 31.28 > Structstrings 1.6.0 117.23 124.00 > TreeSummarizedExperiment 1.6.0 22.60 25.59 > transite 1.8.0 233.41 238.37 > tRNA 1.8.0 43.16 46.70 > tximport 1.18.0 239.30 237.44 > VariantExperiment 1.4.0 24.69 25.32 > xcms 3.12.0 348.71 1576.77 > zinbwave 1.12.0 258.48 257.03 > --------------------------------------------------- > total 4204.97 5439.18 > > Except for the slowdown of xcms tests (which seem to be usingSlowParamparallelization backend), nothing really outstanding here. So with your fix, hopefully overall build times should go back to almost normal. I guess we’ll find out tomorrow. > Thanks for the detective work and for providing a fix. Also thanks@Martin Morganfor the quick patching.

Aaron Lun (18:55:47): > Looks like DMS is really enjoying itself there. 1000 seconds!

Hervé Pagès (18:58:56): > Let’s assume long test run time means good coverage:grin:

Hervé Pagès (19:04:03): > It’d be interesting to see how much cpu time is spent in total on all the unit tests during a full run of the software builds. I wouldn’t be surprise it’s more than 50 hper build machine. Anyways, I’ll keep this for another day.

Hervé Pagès (19:10:06): > Which reminds me that maybe we should start encouraging more people with long test run times to move their heavier tests to the long test builds. The nightly builds are putting a lot of load on the build machines and some machines are reaching their limits. This would help alleviate the load.

2020-11-07

Lluís Revilla (06:06:46): > I went ahead and reported some timings to the testthat team (https://github.com/r-lib/testthat/issues/1223) as I had also seen some issues elsewhere. They are willing to test more on their end but need some information. If anyone has some more information to pinpoint this better please add some comments on the issue.

Aaron Lun (12:52:16) (in thread): > Beginning the slow repair job across all my packages. God, I useI()everywhere.

Aaron Lun (19:14:22): > @Hervé Pagèsthe csaw user’s guide is ready to be a book. How do you want to do this? The tricky part is that the csawUsersGuid package is still built as a workflow package. I’d like to give people a smooth redirect if possible.

Aaron Lun (20:41:04) (in thread): > This should be cleaned out for all my packages.

Hervé Pagès (22:16:25): > I believe the easiest way is that you submit it, like with the previous books. My understanding is that if you putBiocType: Bookin the DESCRIPTION file the SPB will give it more time before it times out. I don’t know how much time and I’m not sure this is implemented yet. I guess we’ll see. Thanks!

Hervé Pagès (22:41:40) (in thread): > Hi Lluis, I think we should wait and see the impact of the BiocParallel fix first. Also I think that the i386 vs x64 story on Windows is so strange that something really fishy must be going on. Or I could have done something really dumb when I extracted these numbers. (In fact I know that I did something dumb at least for basilisk where I switched the i386/after and x64/before values, 717.82 and NA). > Anyways it would probably require a little bit more investigation before we can draw any conclusion. If you have access to a Windows machine maybe you could try to reproduce these observations? This would help a lot. Thanks!

2020-11-08

Aaron Lun (00:16:17): > IT IS DONE

Lluís Revilla (05:24:37) (in thread): > Sorry, I don’t have access to a Windows machine.

2020-11-09

Lluís Revilla (09:17:32) (in thread): > Thanks@Aaron Lunfor the message, hope they fix it.

Peter Hickey (17:06:46) (in thread): > yeah i need to move some to longtests

2020-11-10

Lori Shepherd (13:56:28) (in thread): > That should be implemented on the SPB to get the longer build time – let me know if it doesnt seem to be the case

2020-11-12

Philippe Boileau (15:07:43): > @Philippe Boileau has joined the channel

2020-11-14

Hervé Pagès (12:56:57) (in thread): > The new csawBook is online athttps://bioconductor.org/books/3.13/csawBook/:rocket: - Attachment (bioconductor.org): The csaw Book > ChIP-seq Analysis with Windows. But it works on Mac and Linux too!

Aaron Lun (13:43:54) (in thread): > oh yeah

Aaron Lun (14:35:41) (in thread): > Time to drag the workflows in, and then we can deprecate 2 (two!) other packages.

2020-11-19

Nicholas Cooley (10:51:33): > When we make updates on the current devel branch do we need to but updating the R requirement toR >= 4.1.0?

Kevin Rue-Albrecht (11:04:28) (in thread): > That’s what I did. Otherwise BiocManager complains, I think

Nicholas Cooley (11:06:10) (in thread): > Sounds good, thanks!

Martin Morgan (11:14:00) (in thread): > Many packages haveR >=something less than 4.1; the smallest I saw at a quick glance was diebias, at 1.4.1 (https://bioconductor.org/packages/diebias) I think the only complaint you’ll see is when you use an R feature that R can demonstrably determine requires a particular version of R, usually the version of saved data sets.

2020-11-20

Aaron Lun (16:02:46): > This does not look healthy: > > BiocManager::install("org.Hs.eg.db") > Bioconductor version 3.13 (BiocManager 1.30.10), R Under development (unstable) > (2020-11-09 r79409) > Installing package(s) 'org.Hs.eg.db' > Warning: unable to access index for repository[https://bioconductor.org/packages/3.13/data/annotation/src/contrib](https://bioconductor.org/packages/3.13/data/annotation/src/contrib): > cannot open URL '[https://bioconductor.org/packages/3.13/data/annotation/src/contrib/PACKAGES](https://bioconductor.org/packages/3.13/data/annotation/src/contrib/PACKAGES)' > > The same issue is causing various CI processes to fail, as I promote install warnings to errors in my GHA’s.

Lori Shepherd (16:19:09): > We branched the annotations this morning. The script is still running to propagate them. This should go away once it finishes.

Lori Shepherd (19:26:31): > annotations finished branching and propagating… when I install packages in devel now, I no longer get this warning – it should clear up on tomorrow’s build

Aaron Lun (19:40:24): > yep, my actions are green.

2020-12-02

Konstantinos Geles (Constantinos Yeles) (05:42:08): > @Konstantinos Geles (Constantinos Yeles) has joined the channel

2020-12-09

Aaron Lun (00:37:02): > hooray, the book built again!

Martin Morgan (17:24:30): > …but does the Page Not Found error athttps://bioconductor.org/books/3.13/OSCA/mean that it didn’t (has never) in devel? (or maybe ‘the book’ refers to something else, but still curious…)

Aaron Lun (17:25:11): > the devel builds are broken because they’re waiting for the latest RcppAnnoy

2020-12-11

Charlotte Soneson (08:55:36): > A general, but build-related question: does there exist somewhere a nice tool to help you semi-automatically “guess why your package is failing” on the builders (i.e., is it a problem with your package or with one of its dependencies)? This came up lately as I was looking to figure out whyRhisat2andQuasRwere failing on the devel build machines, and it seems to come down tomakeTxDbFromGFF()not being able to infer the file format properly (alsoGenomicFeaturesis ERRORing), which probably is because of recent changes inrtracklayer/BiocIO(as a side note, everything works fine with fresh installs of all packages locally/on GHA). To automate things a bit, I wrote a small function that will look through the Bioc dependencies of a package and tell you if they are failing, when they were last updated, and what fraction of their direct inverse dependencies are currently failing (available athttps://gist.github.com/csoneson/5a1487cb98043e0835ae38cb39a06cb9) - it’s not very refined and of course there is no guarantee that it will always find the true reason for the build failure, but still…I was wondering if there was something similar, more elaborate, available somewhere, or more in general if others have better ways of tracking things down efficiently:slightly_smiling_face:

Kevin Rue-Albrecht (08:58:37) (in thread): > BiocChallenge materials?:grimacing:

Lori Shepherd (09:12:45) (in thread): > Yes we are aware of the issue with GenomicFeatures::makeTxDbFromGFF@Hervé Pagèswas going to look into fixing this as it does affect several packages. > If you have a script and code that might be useful to the general public it might be worth looking at a PR to BiocPkgTools? > In a related note. I have been working on creating a database to keep track of the build reports. The idea being that the database would be accessible to the public and the ability to query how long a package was failing and potentially these more complicated queries of my package started failing on x day, what dependency had changes in the few days before … its still a work in progress but hope to have out either this cycle of Bioconductor or next.

Charlotte Soneson (09:23:42) (in thread): > Thanks - yes, I was thinking aboutBiocPkgTools, just wasn’t sure if there was anything like this (or better) already somewhere else. Exposing the build report database sounds great!

Hervé Pagès (12:13:52) (in thread): > Note that what makes it really hard to reproduce in that case is that the breaking changes in rtracklayer/BiocIO didn’t propagate yet because rtracklayer’s version wasn’t bumped and the package is ERRORing. I’m waiting for these issues to be addressed before I fixGenomicFeatures::makeTxDbFromGFF()so I can bump the requirement to rtracklayer >= 1.51.1 and my fix has a chance to propagate. I think they are now, today’s report should confirm.

2020-12-12

Aaron Lun (02:43:51): > UUUGGh. devtools depends on usethis which now depends on gert which now depends on a system libgit2 installation.

Aaron Lun (02:44:02): > This is causing build failures for beachmat in prd…. ugh.

Martin Morgan (06:36:33) (in thread): > I think this kind of facility – software to facilitate diagnosis of build system failure – would be a great opportunity for collaborative development, building on@Charlotte Soneson’s code; maybe it could be a topic for a#developers-forum?

Charlotte Soneson (07:27:02) (in thread): > @Kevin Rue-Albrecht,@Martin Morgan- I have submitted a BiocChallenge to get things started (https://github.com/kevinrue/BiocChallenges/pull/25) - perhaps discussion can continue in the linked repo (https://github.com/csoneson/BiocBuildFailureDiagnosis), where I currently moved the code from the gist above.

Martin Morgan (16:05:37) (in thread): > @Kevin Rue-AlbrechtI wanted to mention BiocChallenges in my talk on Monday morning, should I point tohttps://kevinrue.github.io/BiocChallenges/? Is it mentioned on the eurobioc2020 web site? - Attachment (kevinrue.github.io): Challenges for the Bioconductor community > This package hosts challenges contributed by and for the Bioconductor community. It provides functionality to manage, filter, and display challenges as articles of a pkgdown website. Challenges are bite-sized projects led by volunteers, encouraging collaboration and sharing of expertise between contributors.

Kevin Rue-Albrecht (18:10:34) (in thread): > Yes, the compiled pkgdown link (https://kevinrue.github.io/BiocChallenges/) is probably the best place for people to start from. > We haven’t added it on the eurobioc2020 website so far, at least no differently from other short talks, aside from its special timing. - Attachment (kevinrue.github.io): Challenges for the Bioconductor community > This package hosts challenges contributed by and for the Bioconductor community. It provides functionality to manage, filter, and display challenges as articles of a pkgdown website. Challenges are bite-sized projects led by volunteers, encouraging collaboration and sharing of expertise between contributors.

Kevin Rue-Albrecht (18:13:23) (in thread): > The pkgdown website does link back to the github repository, which is the place where contributors will submit new challenges or register their contributions to existing challenges. So, again, the pkgdown website is a good place to start from

Hervé Pagès (20:57:24): > We installed libgit2-dev on the Linux builders on Thursday evening. All usethis-related failures seem to be gone today.

Hervé Pagès (21:01:06) (in thread): > GenomicFeatures::makeTxDbFromGFF()fixed in GenomicFeatures 1.43.3

2020-12-13

Nitesh Turaga (00:45:19) (in thread): > Is this on your github actions? update the images…i’ve added libgit2-dev to both RELEASE_3_12 and devel.

Aaron Lun (00:57:13) (in thread): > No, I was looking at the beachmat BBS reports.

Charlotte Soneson (02:02:43) (in thread): > Thanks Hervé!

FelixErnst (06:31:54) (in thread): > Thanks from me as well!

Huipeng Li (20:36:17): > @Huipeng Li has joined the channel

2020-12-15

FelixErnst (02:55:12): > Hi. I have some weird build warnings for 3.13 on Windows only:http://bioconductor.org/checkResults/devel/bioc-LATEST/Modstrings/tokay2-checksrc.htmlThey seam to be point to an issue for the help pages, which cannot be found during the test for correct installation. This warning also shows up for several other packages (Biostrings,BiocParallel,SingleCellExperiment,etc), so I wonder whether you might have an idea, how to resolve the issue. Thanks for any tips!

2020-12-16

Hervé Pagès (01:40:44): > Hmm.. this actually affects 348 packages! Looks like something has changed in R devel that is causing this.

FelixErnst (02:14:45): > but it is windows specific, isn’t it? or maybe build specific? the tags differ all a bit on the build systems, but what are the chances that the change was introduced afterr79399and fixed byr79432?

Hervé Pagès (04:04:31): > Yes, it looks like a new Windows specific Rd warning, probably unintentional. Maybe they introduced it while they were working on this old one:https://github.com/Bioconductor/MatrixGenerics/issues/19#issuecomment-714591642

FelixErnst (04:40:53): > Looks like it. But it is inconsistent through all of the packages, I have looked at now.MatrixGenericsis affected in 3.12, but not in 3.13. ForBiostringsit is the other way around, which is also the case forModstrings

FelixErnst (04:42:52): > AndMatrixGenericswas changed last on both build on the 27th of October. So I guess there wasn’t a direct fix for it.

Hervé Pagès (05:11:37): > Seems to happen only on packages with aliases that contain<-(e.g.\alias{foo<-}or\alias{foo<-,class-method}).MatrixGenericsdoes not contain such aliases butBiostringsandModstringsdo:grimacing:

FelixErnst (05:15:57): > <is not allowed in Windows file names:expressionless:

FelixErnst (05:16:06): > Is that it?

Hervé Pagès (12:03:53): > Note sure why these files would be needed. It’s not that the help system needs to create one HTML file per alias.

FelixErnst (12:06:47): > also not during the build process? I have never seen the html files, but they are probably wrapped during build into therd*files found in the help folder.

Hervé Pagès (13:25:33): > <!channel>Any volunteer for reporting this to the R-devel mailing list? You’d need access to a Windows machine and make sure you can reproduce there first, or I’m not responsible for what happens to you:grin:

FelixErnst (13:26:32): > fyi: I am out, since I switched to WSL2 at the expense of a running Vbox installation

Mike Smith (14:25:02): > Is anyone seeing in on another CI platform like GitHub actions? I’m not sure I have any help files with\alias{foo<-}in, or what specific R-devel version is available on there. If not I can fire up my Windows machine tomorrow and see if I can recreate with one of the packages listed here.

2020-12-17

Charlotte Soneson (04:02:02): > iCOBRAis one of the packages with these warnings (http://bioconductor.org/checkResults/devel/bioc-LATEST/iCOBRA/riesling1-checksrc.html), but they’re not showing up in GHA (https://github.com/csoneson/iCOBRA/runs/1536280612?check_suite_focus=true). On the other hand, google pointed me to a couple of packages giving warnings (~a month ago) on AppVeyor, e.g.https://ci.appveyor.com/project/PhilBoileau/scPCA(this package is not among those with the warnings on the build machines now, though).:thinking_face:(in addition to names with<, redirection here seems to fail also for names containing:).

Nicholas Cooley (10:49:07): > I don’t know if this is the right place for this question, but what’s the approximate delay between updating a package on the devel branch, and it updating on the build report? For some reason I thought devel built every evening?

Lori Shepherd (10:54:07): > it builds every evening that is correct. there is some information on timing at the top of this pagetroubleshooting. Depending on when you pushed your commit, if it is after when we have pulled fromgit.bioconductor.orgfor the day then it could take up to 36 hours to appear on the build report as it would not appear on the next day’s report but the day after that

Nicholas Cooley (10:56:28): > Oh great! Thanks!

Manojkumar Selvaraju (11:12:07): > @Manojkumar Selvaraju has joined the channel

Mike Smith (16:10:34) (in thread): > Just to follow up on this I installed the latest R-devel for Windows (r79643) and clonedModstringsfrom the Bioc git repo. RunningR CMD checkdoesn’t produces the errors seen in the build system.

FelixErnst (17:54:12) (in thread): > Thanks!

Hervé Pagès (18:19:29): > It’s an installation problem. As you can see, R does a lot of weird and noisy things with the help pages during installation on Windows:https://bioconductor.org/checkResults/3.13/bioc-LATEST/Modstrings/riesling1-install.htmlDo you see the “Rd warning: cannot open file …” lines there? They kind of get lost in the middle of the crazy output but they are here. If you capture this output in a file called, say,Modstrings.install-out.txt, then you can callR CMD checkwith--install=check:Modstrings.install-out.txtand--library=\path\to\your\R\library, and you should see the warning. The build system uses the--install=check:...+ library=...trick to avoid an additional installation of the package and save time (they use the same trick for the CRAN checks, I got it from them). It’s unfortunate that this is not 100% semantically equivalent to running a standardR CMD checkthough, this is the first discrepancy I see. Kind of an orthogonal issue. But yeah, doesn’t help folks reproduce the build/check results. > Gosh, reporting this to the R-devel mailing list is not going to be an easy one!:worried:

2020-12-18

Mike Smith (03:04:09): > Thanks for the clarification. I’ll try and get it running through with those settings and report back. One thing I did check was the html folder for the package in the library, and theseqtype<-,ModString-method.htmlfile and equivalent were not present. I assumed that was because it wasn’t trying to make them anymore, but perhaps it was still failing and I wasn’t seeing the error.

Mike Smith (10:40:07): > Ok, I can confirm that following your approach reproduces the error forModstringswithr79643Is that--install=check:option even documented? I don’t see it as an option in the output toR CMD check --help. Also, I was following the commands show on the Summary section of the build reports, but I don’t see whereModstrings.install-out.txtis created. For this test I ended up copy/pasting from the terminal into a text editor and saving with that name. Is there an automated step I’m not seeing?

Hervé Pagès (12:40:59): > Last time I checked the--install=check:option was not documented, unfortunately. > > The build system creates theModstrings.install-out.txtfile earlier, during the INSTALL step, by capturing the output ofR CMD INSTALL. FWIW the build system is written in Python and all theR CMDcommands used by the various stages of the builds are fired from Python via thesubprocessmodule. This allows fine control of the processes including capture of their output. When working interactively you can capture the output by redirecting it to a file with>: > > R CMD INSTALL Modstrings >Modstrings.install-out.txt > > Yes, as crazy as this might sound, this works on Windows too! (in a PowerShell) > > Anyway, all this to say that the command displayed in the Summary section is unfortunately not standalone as it relies on things produced earlier. Furthermore, in some cases (e.g. command used by the BUILD BIN stage on Mac), a simple copy/paste of the command won’t work, even if you have all the required build products, because the full command uses/Users/biocbuild/BBS/utils/build-universal.shto perform some Mac-specific tricks in order to produce a binary that is distributable. (These tricks are from Simon Urbanek and he uses them to produce the Mac binary packages provided by CRAN.) You would need to clone the BBS repo first to have access to thebuild-universal.shscript. It’s still super useful, at least for us (build system maintainers), to have these commands displayed on the report for when we need to go on a build machine to try and reproduce the things we see on the report.

2020-12-22

Aaron Lun (15:40:59): > Looks like another set of impossible basilisk errors on Reisling. Diagnostics suggest that it thinks it’s a 32-bit architecture:http://bioconductor.org/checkResults/devel/bioc-LATEST/basilisk/riesling1-checksrc.html(see how it’s pulling from thewin-32conda channel) > > In contrast to tokay, which correctly thinks its in 64-bit mode:http://bioconductor.org/checkResults/devel/bioc-LATEST/basilisk/tokay2-checksrc.html(see thewin-64references)@Hervé Pagèsif you can do me a favor, it should be possible to confirm this by looking at the nature of the installer at: > > list.files(dirname(basilisk.utils::getExternalDir())) > > I get, for example,Miniconda3-py37_4.8.3-MacOSX-x86_64.sh; what does this say on tokay vs reisling?

Hervé Pagès (21:51:43): > On tokay2: > > > library(basilisk.utils) > > list.files(dirname(basilisk.utils::getExternalDir())) > [1] "00LOCK" > [2] "1.3.3" > [3] "Miniconda3-py37_4.8.3-Windows-x86_64.exe" > > On riesling1: > > > library(basilisk.utils) > > list.files(dirname(basilisk.utils::getExternalDir())) > [1] "00LOCK" > [2] "1.3.3" > [3] "Miniconda3-py37_4.8.3-Windows-x86.exe" > [4] "Miniconda3-py37_4.8.3-Windows-x86_64.exe" >

Aaron Lun (21:53:41): > ooooh

Aaron Lun (21:54:10): > can you get the timestamps on the two installers for reisling?

Aaron Lun (21:56:24): > The thing I don’t understand is, I’ve turned off win32 builds.

2020-12-23

Hervé Pagès (01:44:18): > > PS C:\Users\biocbuild\AppData\Local\basilisk> C:\rtools40\usr\bin\ls.exe -al > total 105348 > drwxr-xr-x 1 biocbuild None 0 Dec 21 16:52 . > drwxr-xr-x 1 biocbuild None 0 Dec 8 18:54 .. > -rw-r--r-- 1 biocbuild None 0 Nov 7 21:32 00LOCK > drwxr-xr-x 1 biocbuild None 0 Dec 21 19:49 1.3.3 > -rwxr-xr-x 1 biocbuild None 50599000 Dec 16 16:58 Miniconda3-py37_4.8.3-Windows-x86.exe > -rwxr-xr-x 1 biocbuild None 57256056 Nov 26 17:31 Miniconda3-py37_4.8.3-Windows-x86_64.exe >

Aaron Lun (11:37:08): > Hm. HMM….

Aaron Lun (11:37:16): > I am bemused.

Hervé Pagès (12:03:12): > I see that Rcwl started to use basilisk on Dec 14 (commit 9b6ba9d7e2979b9f5a7e4ca738ddd0ddbc443a85) but at the same time they did not addUnsupportedPlatforms: win32to their .BBSoptions file (they had a .BBSoptions withUnsupportedPlatforms: winbut removed it). Could it be that this somehow triggered the installation of 32-bit miniconda?

Hervé Pagès (12:04:38): > Wonder why this wouldn’t have the same effect on tokay2 though…

Aaron Lun (12:08:57): > Hm. Maybe. Hm…

Aaron Lun (12:09:41): > A bit surprising that would have happened; basilisk should have been isntalled before its dependencies, so it should have created its own 64-bit build first.

Hervé Pagès (12:13:18): > Want me to removeC:\Users\biocbuild\AppData\Local\basiliskon riesling1 in the meantime?

Aaron Lun (12:13:24): > yes please.

Aaron Lun (12:13:33): > I’ll have to think about what on earth happened

Hervé Pagès (12:16:02): > done

Aaron Lun (12:16:05): > thansk

Hervé Pagès (12:28:36): > Since I was on riesling1 I thought I would just try: > > R --arch x64 CMD INSTALL --no-multiarch basilisk > > followed by: > > R CMD INSTALL --merge-multiarch Rcwl > > The 2nd command failed with: > > ... > Error: package or namespace load failed for 'Rcwl': > .onLoad failed in loadNamespace() for 'Rcwl', details: > call: NULL > error: one or more Python packages failed to install [error code 1] > Error: loading failed > Execution halted > ERROR: loading failed > * removing 'D:/biocbuild/bbs-3.13-bioc/R/library/Rcwl' > > like on the build report. But before this it managed to recreate and repopulateC:\Users\biocbuild\AppData\Local\basiliskwith: > > PS C:\Users\biocbuild\AppData\Local> C:\rtools40\usr\bin\ls.exe -al basilisk > total 49424 > drwxr-xr-x 1 biocbuild None 0 Dec 23 12:21 . > drwxr-xr-x 1 biocbuild None 0 Dec 23 12:21 .. > -rw-r--r-- 1 biocbuild None 0 Dec 23 12:21 00LOCK > drwxr-xr-x 1 biocbuild None 0 Dec 23 12:21 1.3.3 > -rwxr-xr-x 1 biocbuild None 50599000 Dec 23 12:21 Miniconda3-py37_4.8.3-Windows-x86.exe >

Hervé Pagès (12:32:26): > Just to be clear, the first command (R --arch x64 CMD INSTALL --no-multiarch basilisk) didn’t recreateC:\Users\biocbuild\AppData\Local\basilisk. Only the 2nd command did (and repopulated it).

Aaron Lun (13:00:47): > Hm… very interesting.

Aaron Lun (13:01:02): > Oh damn! I see.

Aaron Lun (13:01:26): > All packages are installed first before CHECK. And, right, the install won’t trigger the creation of a basilisk environment.

Aaron Lun (13:02:22): > In fact, installation should not create an environment foranybasilisk client, unlessBASILISK_USE_SYSTEM_DIRis set.

Aaron Lun (13:02:35): > sounds like Rcwl is doing something not quite good.

Hervé Pagès (13:08:54): > Not good that one of your reverse deps can mess up your own stuff.

Aaron Lun (13:12:50): > Well, it’s kind of like that time when a package overwrote the global Makevars. Not a lot I can do about it.

Aaron Lun (13:13:05): > Guess I could cut off win32 support within the installation machinery, which would force the error to occur in Rcwl instead.

Aaron Lun (13:17:56): > Hm. That seems like the best approach. I don’t even know how much of it works properly on win32 anyway.

Aaron Lun (13:21:16): > well, while I figure this out, I”ll mention that the rank badges are borked.

Hervé Pagès (13:25:27): > That’s because they rely on the download stats which are broken at the moment. Should be back before the end of the week.

2021-01-05

Aaron Lun (12:32:57): > @Hervé Pagèsis it much effort to get landing pages for the books? It would be really nice in the book’s introduction if I could tell people to doBiocManager::install("OSCA")and get all the packages.

Lori Shepherd (12:40:08): > @Hervé PagèsWe can discuss some of the specifics off line for what the website code needs for the books – Without looking into it in full detail, I don’t think it should be too complicated but would need a few files generated that I don’t think we currently have

Hervé Pagès (13:19:20): > Landing pages and makingBiocManager::install("OSCA")work are 2 different things. For the latter we only need to set up a new package repository for book source tarballs (like we already have for software, data-annotation, data-experiment, and workflows), and to add the new repo toBiocManager::repositories(). Landing pages can come later.

Aaron Lun (13:20:29): > Right. I was just thinking of another way of people to be able to discover that they could actually doinstall("OSCA").

Hervé Pagès (13:22:33): > Sure, a landing page can help, granted that people are able to discover it:smiley:. The intro in the book should definitely tell them aboutBiocManager::install("OSCA").

Hervé Pagès (13:23:02): > Once it’s ready.

Dan Bunis (13:24:58): > ^ On the development side, not realizing that I could install (via git clone, but I think still relevant) to get relevant dependencies did hamper my initial attempt to contribute to Aaron’s planned (?) SCGallery.

Hervé Pagès (14:23:28): > 3.12/books and 3.13/books repos are ready. The OSCA book and its deps can be installed with: > > install.packages("OSCA", dependencies=TRUE, repo="[https://bioconductor.org/packages/3.12/books](https://bioconductor.org/packages/3.12/books)") > > Next thing is to add the books repo toBiocManager::repositories(). I’m preparing the PR.

Aaron Lun (14:24:08): > very nice

Aaron Lun (14:24:45): > Now that we’ve had a full release cycle of reliability, I might also finally scrap the trojan.

Hervé Pagès (14:26:22): > oh, you still have the trojan in the simpleSingleCell workflow?

Aaron Lun (14:26:42): > yes, in OSCA and in the SingleRBook. (simpleSingleCell is effectively dead.)

Aaron Lun (14:26:55): > By comparison, csawBook is the first non-trojan book.

Hervé Pagès (14:27:52): > I’m kind of lost, but it doesn’t matter.

Hervé Pagès (14:41:18): > Done:https://github.com/Bioconductor/BiocManager/pull/89

2021-01-06

Matt Stone (13:15:03): > Hello, have there been any recent changes to the malbec1 build system? I received a message this morning that today’s build timed out, although I haven’t pushed any updates to release since the 3.12 release. > > I’ll look into the code on my end, but was just curious if there was anything in the malbec1 environment that I should be aware of. Thanks!

Hervé Pagès (15:58:16): > I’m not aware of any change made to malbec1. Maybe something has changed in one of the packages you depend on? With 148 (direct and indirect) dependencies, this is the kind of stuff that happens, unfortunately. Not that you also get a TIMEOUT on nebbiolo1 (release) and malbec2 (devel): > * https://bioconductor.org/checkResults/3.12/bioc-LATEST/BayesSpace/ > * https://bioconductor.org/checkResults/3.13/bioc-LATEST/BayesSpace/ > which suggests that the issue is likely to be reproducible on any recent Ubuntu system.

Hervé Pagès (17:59:02): > Hmm.. actually it’s not. Can’t reproduce this on my laptop (Ubuntu 20.10):thinking_face:Let’s wait a couple more days and see what happens with the builds.

2021-01-20

Hervé Pagès (00:45:59): > Or maybe the issue is reproducible on any Ubuntu system prior to Ubuntu 20.10. Here is what I found so far: BayesSpace’s unit tests alone take about 40 min to complete on malbec2 vs < 30s on merida2 (Mac builder) and on my laptop (Ubuntu 20.10). One of the tests callsxgb.train()from thexgboostpackage. Looks like this call takes a fraction of a second on all machines except on our Linux build machines where it’s hundreds of times slower: > > library(xgboost) > packageVersion("xgboost") > # [1] '1.3.2.1' > X.ref <- matrix(runif(2000, min=-7, max=7), 100, 10) > Y.ref <- matrix(runif(2000, max=11), 20, 100) > rownames(X.ref) <- colnames(Y.ref) <- paste0("spot_", 1:100) > colnames(X.ref) <- paste0("PC", 1:10) > rownames(Y.ref) <- paste0("gene_", 1:20) > gene <- "gene_4" > set.seed(100) > train.index <- sample(1:ncol(Y.ref), 64) > data.train <- xgb.DMatrix(data=X.ref[train.index, ], > label=Y.ref[gene, train.index]) > data.test <- xgb.DMatrix(data=X.ref[-train.index, ], > label=Y.ref[gene, -train.index]) > watchlist <- list(train=data.train, test=data.test) > > system.time(fit.train <- xgb.train(data=data.train, max_depth=2, watchlist=watchlist, eta=0.03, nrounds=500, objective="reg:squarederror", verbose=FALSE)) > > The call toxgb.traintakes: > * 118.35s on malbec1 (Ubuntu 18.04, gcc/g++ 7.5.0) > * 77.86s on malbec2 (Ubuntu 20.04, gcc/g++ 9.3.0) > * 0.23s on merida2 (macOS Mojave, Apple clang version 11.0.0) > * 0.18s on my laptop (Ubuntu 20.10, gcc/g++ 10.2.0) > xgboost contains C++14 code and I wonder if there is an issue with the binary code generated by gcc/g++ 9.3.0 or earlier on Linux. > > Also it looks like a new version ofxgboost(version 1.3.1.1) appeared on CRAN on Jan 5th and another one (version 1.3.2.1) just 2 days ago (Jan 18). Here are the timings on malbec2 with these older versions: > * 83.34s with xgboost 1.2.0.1 > * 73.33s with 1.1.1.1 > So there is still the mystery of why we only started to see the issue recently forBayesSpaceand why the unit tests take 40 min. After all they contain only one call toxgb.train(), and, even if the call is 100x slower on malbec2 compared to the Windows or Mac builders, it takes only a couple of minutes.

Aaron Lun (00:47:51): > Hm. Very curious. scDblFinder uses xgboost as well, though it does not seem appreciably slower.

Hervé Pagès (00:57:07): > Unfortunately I can’t really spend more time on this but I suspect that the issue is unrelated to some weird configuration of our Linux builders. It would be super useful if someone could confirm this by reproducing the BayesSpace issue on an Ubuntu <= 20.04 system. Can someone try this on the official docker image?

Aaron Lun (01:12:28): > Well, I guess I owe you one for the OSCA stuff, so I can do it. > > > system.time(fit.train <- xgb.train(data=data.train, max_depth=2, watchlist=watchlist, eta=0.03, nrounds=500, objective="reg:squarederror", verbose=FALSE)) > user system elapsed > 1.813 0.000 0.189 >

Aaron Lun (01:13:14): > This is on Docker, Ubuntu 18.04.5.

Hervé Pagès (01:56:09): > Bummer. Can you time the full BayesSpace test suite? > > library(testthat) > library(BayesSpace) > system.time(test_check("BayesSpace")) > > Thanks!

Aaron Lun (02:32:48): > Hm. On hindsight I should have started building off one of my existing single-cell docker images.

Aaron Lun (03:00:12): > 16 seconds.

Aaron Lun (03:00:20): > > > library(testthat) > > library(BayesSpace) > > system.time(source("testthat.R")) > [ FAIL 0 | WARN 0 | SKIP 0 | PASS 43 ] > user system elapsed > 95.579 3.710 16.516 >

Mike Smith (04:25:43): > I see similar performance to Aaron on my Linux Mint 20 machine: > > > system.time(source("testthat.R")) > [ FAIL 0 | WARN 0 | SKIP 0 | PASS 43 ] > user system elapsed > 50.003 0.354 31.665 >

Mike Smith (04:28:17): > The massive difference between theuserandelapsedtimes in the build report makes me wonder if it’s doing something unexpectedly parallel on the build machines, which we don’t see on our desktop runs. > > * checking examples ... OK > Examples with CPU (user + system) or elapsed time > 5s > user system elapsed > enhanceFeatures 10749.140 20.720 280.100 > > Could it be trying to use all cores and then running into some other bottleneck like memory?

Mike Smith (05:30:39): > The man page forxgb.trainsays: > > Parallelization is automatically enabled if OpenMP is present. Number of threads can also be manually specified vianthreadparameter. > ~~~For me that argument doesn’t seem to do anything. Whatever I set it to I can clearly see all 4 cores being used in ~~~~~htop~~~~~~. Likewise if I try setting the ~~~~~~OMP_NUM_THREADS~~~~~~ environment variable.~~~~Turns out if you actually read the instructions and usenthreadrather thannthreadsit works rather better! Also settingexport OMP_THREAD_LIMIT=1seems to have the desired effect without modifying the code.

Hervé Pagès (10:47:08): > Excellent. Thx Mike. I’m going to give this a try…

Hervé Pagès (10:58:03): > Guess what, now I’m on malbec2 again and the exact same call toxgb.train()takes 0.224s instead of 77.86s when I tried it yesterday night. Difference is that yesterday the daily builds were running but now they are done. Seems likexgb.train()’s performance can be deeply affected by the system load, in a totally insane way.

Hervé Pagès (11:00:27): > On Linux only (builds are also heavily parallelized on the other platforms and everything is fine there). I mean, the other platforms don’t support OpenMP so, yeah, another clue that this is OpenMP related.

Hervé Pagès (11:42:04): > Anyways,OMP_THREAD_LIMIT=2now set on malbec2. Let’s see how it goes.

Alan O’C (11:45:00): > I think it may be because of testthat parallelisation combining badly with openmp. Any time I’ve tried to combine fork-based parallelism (eg, mclapply) with openMP R seems to hang indefinitely (or at least so long I won’t wait). Though if you tested manually and saw the same, possibly not

Hervé Pagès (11:48:35): > Do you know how to turn off testthat parallel execution?

Hervé Pagès (11:50:11): > I could try to do this globally for the builds.

Alan O’C (11:52:27): > Not a clue

Hervé Pagès (12:06:49): > Found it:TESTTHAT_CPUS. > > Anyways I’m not sure testthat runs the tests in parallel by default. Looks like you need to putConfig/testthat/parallel: truein your DESCRIPTION file for that:https://testthat.r-lib.org/articles/parallel.html - Attachment (testthat.r-lib.org): Running tests in parallel > testthat

Hervé Pagès (12:08:46): > And I don’t see that any BioC package is using that so we’re good.

FelixErnst (12:36:04): > Just an fyi: Around the 5th of Nov.@Aaron Lundid some testing for scran (?) and I also had some weird run time issue during package testing locally. One comment stuck out testthat is doing some weird environment wrangeling.

Alan O’C (12:36:37) (in thread): > Cool, good to know! I might be creating associations where there are none

FelixErnst (12:36:55): > That conversation was here in this channel. Maybe that contributes to weird run time behavior as well

Hervé Pagès (12:41:10): > so it’s@Aaron Lunfault, again:zany_face:

Aaron Lun (12:41:26): > I DENY EVERYTHING!

Hervé Pagès (12:41:48): > You have the right to remain silent… for now

Aaron Lun (23:21:39): > Hm. Is someone filling up/tmpon the build system?

Aaron Lun (23:24:51): > oh wait. It’s probably a difference in our pandoc versions.

2021-01-21

Hervé Pagès (00:16:19): > It worked! WithOMP_THREAD_LIMIT=2now in the Renviron.bioc file on malbec2, BayesSpace’s just passed CHECK in 310.7 sec. on this machine, as part of the nightly devel builds. Just committed the change to the BBS code and will deploy to the other Linux builders.

Matt Stone (10:06:55): > Thank you all so much for looking into this, and thanks Hervé and Mike for the final fix! Sorry it ended up being so involved.

Matt Stone (10:07:05): > I’m catching up on this thread - I can specifynthread=1in our unit test to avoid this also becoming an issue locally for users building/testing on Linux. Are there any other changes I can/should make?

Mike Smith (10:20:46) (in thread): > I was thinking about what the best behaviour is for this. I’m not a fan of code that defaults to using everything available, mostly because it leads to weird situations like this, or unexpectedly hammers shared computing environments like cluster nodes. > > Personally I’d advocate adding annthreads(I’m sticking to my plural here!) argument to your own functions that callxgb.train(), with a default of 1, and then passing that through. Another option might be to usegetOption("Ncpus", 1L), then if someone has set a number of cores for R to use you’ll obey it and otherwise play it safe.

2021-01-22

Annajiat Alim Rasel (15:41:12): > @Annajiat Alim Rasel has joined the channel

2021-01-25

Hervé Pagès (19:29:10): > @Aaron LunI can see that the 4 OSCA.* subbooks made their way to the devel builders. Let’s see what happens on the next build (tomorrow).

Aaron Lun (19:29:19): > fingers crossed

Aaron Lun (19:29:36): > rebook also rebuilt properly after I sorted out the differences between mine and BBS’s pandoc

Aaron Lun (19:29:45): > so we should, in theory, be in a good position.

Hervé Pagès (19:30:24): > shhh, don’t jinx it

Aaron Lun (19:31:09): > ha

Hervé Pagès (19:31:58): > so many bad things can happen e.g. I just found out that my latest changes to the BBS code broke the release builds

Hervé Pagès (19:32:37): > trying to repair now

2021-01-26

Hervé Pagès (01:34:27): > I started the book builds earlier on rex3 (a new Linux builder, not included on the official report yet) and they’re done. They look good:https://bioconductor.org/checkResults/3.13/rex3/books-LATEST/

Aaron Lun (01:38:50): > hot damn, it’s beautiful

Hervé Pagès (19:08:40): > Unfortunately, not so much on~malbec1~malbec2:https://bioconductor.org/checkResults/3.13/books-LATEST/With the new modularization, the book builds take even longer and overlap with the software builds. For example the “there is no package called ‘BiocSingular’” error for OSCA.workflows is because the software builds were reinstalling BiocSingular at that moment. I think I’m going to replace malbec2 with rex3 for the book builds. The latter is much more powerful. It builds and checks all the software packages in about 4h vs 15h for the former!

Aaron Lun (19:09:54): > yes, the modularity comes at the cost of having to repeat a few things. Wonder why it took 15 hours, though, it doesn’t repeat all that much.

Hervé Pagès (19:11:00): > 15h is for the software builds. The book builds on malbec2 take about 5-6 h or something like that.

Aaron Lun (19:11:09): > oh, sorry, wasn’t looking properl.

Aaron Lun (19:11:23): > hm. wonder where those extra 2 horus come from. Oh well.

Aaron Lun (19:12:02): > Is rex3 a replacement for malbec2?

Hervé Pagès (19:15:08): > Because I limit the nb of cpus used for the book builds to 3 on malbec2. With the modularization there are more packages to build (7 instead of 3) so they have to wait their turn like at the post office. This is why OSCA.workflows was so late and ran into a conflict with the software builds. We won’t have any of these problems on rex3. Yes, the plan is to replace malbec2 with rex3 at some point.

2021-01-27

Hervé Pagès (02:22:43): > Done:https://bioconductor.org/checkResults/3.13/books-LATEST/(still need to work on propagation from rex3 tobioconductor.org)

Hervé Pagès (13:14:31): > @Aaron LunOK so right now I’ve modified the deployment script to publish the content ofinst/doc/book, and to usevignettes/book/docsonly as a fallback: > > biocadmin@rex3:~/propagation-pipe/3.13$ ./deploy-books.sh > Deploying csawBook book ... (tarball has no 'csawBook/inst/doc/book' folder, deploying 'csawBook/vignettes/book/docs' instead) ... OK > Deploying OSCA.advanced book ... OK > Deploying OSCA.basic book ... OK > Deploying OSCA.intro book ... OK > Deploying OSCA.workflows book ... OK > Deploying SingleRBook book ... (tarball has no 'SingleRBook/inst/doc/book' folder, deploying 'SingleRBook/vignettes/book/docs' instead) ... OK > > Maybe we should standardize a little? > Also is it healthy to deploy some subbooks but other no because they failed? Aren’t we going to end up with published book parts that are not in sync with each others?

Aaron Lun (13:15:14): > Yes, we should standardize, I’ll flip the other two to useinst/doc/booklater.

Aaron Lun (13:17:04): > If one book fails, any other book that depends on its content (either viarebook::linkorrebook::extractFromPackage) will fail as well, so we’ll end up with a whole series of failures for the network of affected subooks… as it should be.

Hervé Pagès (13:22:25): > hmm.. so IIUC you’re saying that the responsibility of making sure that the book components that get published remain in sync is transferred to the book maintainer. Also what about the main package (the one to rule them all). For example right now OSCA fails so didn’t get published even though some subbooks were published. Is that ok?

Aaron Lun (13:24:08): > In general: yes, it is okay. I mean, it’s the same problem as what happens when a package doesn’t get propagated but its dependencies do.

Hervé Pagès (13:24:34): > is it?

Aaron Lun (13:24:37): > In this specific case: I just haven’t updated OSCA yet. Once I do, it’ll be a shell that redirects to all of the individual subbooks (plus some book-related sundries).

Aaron Lun (13:29:04): > Why not? Plenty of cases where stuff like, e.g., scran fails but lower-level packages like SCE build correctly. At least people can look at the latest content for the subbooks even if the latest version of the dependent books are not fully propagated.

Aaron Lun (13:30:07): > The only problem that the static links in the older versions of some books may no longer be valid, but this can only be fixed by propagating a new version.

Aaron Lun (13:30:51): > And withholding updates for the entire network of books until it all builds is too close to our previous monolithic model. Sometimes books fail for no real reason e.g., EHub connection failures.

Hervé Pagès (13:36:17): > I guess I had a misconception of what you really intended to do with the book split. I kind of assumed it was an internal thing only and that it wouldn’t affect the final book layout (still a big monolithic book). But it seems that what you really meant was having a collection of individual books that cross-link each others. So yeah, if that’s the case, the situation is very similar to what happens with packages in general.

Aaron Lun (13:37:15): > Yeah. With some careful linking, we can really make it feel like the same book, even though it is internally all separate.

Aaron Lun (13:37:31): > will have to spend the weekend ironing out the kinks in the visuals

Hervé Pagès (13:39:12): > Semantic question: are we still going to refer to it as the OSCA book or as the OSCA books?

Aaron Lun (13:39:38): > good q

Aaron Lun (13:40:14): > I was hoping that if the spatial transcriptomics book takes off, it could fall under the general OSCA umbrella, and we could refer to it as a series of books.

Aaron Lun (13:41:00): > because then it would be more of a collection, written by different people, with different datasets and concepts, etc.

Hervé Pagès (13:41:43): > e.g. if someone says I found error blah blah in the OSCA book version x.y.z, we don’t really know what version they’re talking about because we don’t know the individual versions of their subbooks. unless they’ve also installed them and report their sessionInfo() but that’s unlikely.

Aaron Lun (13:42:39): > hm.

Hervé Pagès (13:44:44): > In other words I wonder if it still makes sense to try to make the thing feel like the same book. Won’t that be confusing, especially if each subbook is kind of standalone now?

Aaron Lun (13:48:47): > The key with the “same book” feel is that the links between content are seamless. (In all other respects, the chapters are separate HTMLs anyway, so it doesn’t make a difference there.)

Aaron Lun (13:49:25): > In this case,rebook::linkwill preface each link with the prefix of the book, e.g.,Basic Chapter 5,Advanced Figure 2.1,Workflow Section 3.2.1.

Aaron Lun (13:49:49): > Hopefully that should “namespace” the content within each subbook.

Aaron Lun (13:51:14): > THe only other aspects of the “same book” feel relate to the formatting, choice of colors, etc. And I don’t mind having the same theme for all these books, we might as well.

2021-01-29

Peter Hickey (00:43:28): > i’m trying to make use oflongtestsforDelayedMatrixStats. You can see my initial attempt athttps://github.com/PeteHaitch/DelayedMatrixStats/commit/c65ed73c0785de9c6e2144380a917d53185b3580

Peter Hickey (00:44:05): > but i don’t think they’re actually being run, seehttps://bioconductor.org/checkResults/devel/bioc-longtests-LATEST/DelayedMatrixStats/where there’s notest_check("DelayedMatrixStats")output that I expected

Peter Hickey (00:46:48): > looking at some other packages that uselongtests(DropletUtilsandbeachmat) it appears that they actually have a separate, distinct suite of tests inlongtests/testthat(https://github.com/MarioniLab/DropletUtils/tree/master/longtests)

Aaron Lun (00:46:51): > yeah, and it takes 80 seconds, that can’t eb right

Peter Hickey (00:47:18): > but i just want to re-run my existing test suite but withDelayedArray::setAutoBlockSize(8)

Peter Hickey (00:47:42): > any suggestions of a good way to do this without copying/duplicating the entire test suite inlongtests?

Aaron Lun (00:48:20): > aretests/present in the installation directory?

Aaron Lun (00:49:18): > or you could write aconfigureto copy alltests/intoinst/during package install, and then havelongtestsretrieve content from there.

Aaron Lun (00:50:34): > not sure whether that would work, though, I don’t really know the state of the directory when R cmd check runs.

Aaron Lun (00:50:57): > worst case, move all tests intoinst/and then have bothtests/andlongtestsrefer to them.

Peter Hickey (00:52:34) (in thread): > I think only if run asR CMD INSTALL --install-tests

Aaron Lun (00:53:23): > actually, you could try to see whether you can refer to e.g.test_dir("../tests/testthat").

Peter Hickey (00:53:39) (in thread): > I’ve not kept up with changes totestthatbuthttps://testthat.r-lib.org/news/index.html#minor-improvements-and-bug-fixessuggestsinst/testsis no longer supported - Attachment (testthat.r-lib.org): Changelog

Aaron Lun (00:54:02): > I see you’re runningtest_check("DelayedMatrixStats"). I don’t think that’ll work, because the new test directory islongtests/; R CMD check no longer knows anything abouttests/.

Aaron Lun (00:54:19) (in thread): > as in, manually refer to it withtest_dir.

Hervé Pagès (00:55:57): > Thelongtestsfolder is mandatory. Maybe try with a symlink to..\tests\testthat?

Hervé Pagès (00:58:36): > On Windows the build system automatically transforms any symlink into the referent file/dir.

Aaron Lun (00:59:09): > I was just going to ask what happens on windows

Aaron Lun (00:59:18): > How does that work? A copy?

Hervé Pagès (00:59:36): > there’s an rsync option for that

Aaron Lun (01:00:57): > ah I thought it did some windows equivalent to hard-linking

Aaron Lun (01:01:20): > but it seems the reality is much more mundane

Hervé Pagès (01:05:43): > but the nice thing about this rsync option is that it works everywhere, whether the destination system supports hard/soft links or not

Aaron Lun (01:06:19): > sure

Peter Hickey (01:06:39) (in thread): > I’m giving that a try on my macbook. if it looks promising i’ll push to bioc and let it try over the weekend

Hervé Pagès (01:08:24): > and it’s not that we expect the content oftests\testthatto be huge so copying is not really a concern. That’s what yourconfiguresolution was going to do after all.

Hervé Pagès (01:10:05): > another approach is to use RUnit:smile:

Aaron Lun (01:10:34): > I certainly prefer RUnit’s lack of namespace mangling

Aaron Lun (01:10:45): > but the fact that everything needs to be inside a function is a pain

Aaron Lun (01:11:37): > well, the creation of the function itself is not the problem; it’s the fact that

Aaron Lun (01:11:39): > oh call

Peter Hickey (01:20:42) (in thread): > doesn’t seem to have ‘noticed’ the symlink?

Peter Hickey (01:20:42) (in thread): > > (base) cd ~/GitHub/DelayedMatrixStats/longtests > (base) ln -s ../tests/testthat . > (base) cd ~/GitHub > > (base) Peter@Peters-MacBook-Pro-2 GitHub $ R CMD INSTALL DelayedMatrixStats > * installing to library '/Library/Frameworks/R.framework/Versions/4.1/Resources/library' > * installing **source** package 'DelayedMatrixStats' ... > **** using staged installation > **** R > **** inst > **** byte-compile and prepare package for lazy loading > **** help > ***** installing help indices > **** building package indices > **** installing vignettes > **** testing if installed package can be loaded from temporary location > **** testing if installed package can be loaded from final location > **** testing if installed package keeps a record of temporary installation path > * DONE (DelayedMatrixStats) > > (base) Peter@Peters-MacBook-Pro-2 GitHub $ R CMD build --keep-empty-dirs --no-resave-data DelayedMatrixStats > * checking for file 'DelayedMatrixStats/DESCRIPTION' ... OK > * preparing 'DelayedMatrixStats': > * checking DESCRIPTION meta-information ... OK > * cleaning src > * installing the package to build vignettes > * creating vignettes ... OK > * cleaning src > * checking for LF line-endings in source and make files and shell scripts > * checking for empty or unneeded directories > WARNING: directory 'DelayedMatrixStats/longtests' is empty > WARNING: directory 'DelayedMatrixStats/src' is empty > WARNING: directory 'DelayedMatrixStats/tests/testthat/_snaps' is empty > * building 'DelayedMatrixStats_1.13.4.tar.gz' > > (base) Peter@Peters-MacBook-Pro-2 GitHub $ R CMD check --test-dir=longtests --no-stop-on-test-error --no-codoc --no-examples --no-manual --ignore-vignettes --check-subdirs=no DelayedMatrixStats_1.13.4.tar.gz > * using log directory '/Users/Peter/GitHub/DelayedMatrixStats.Rcheck' > * using R Under development (unstable) (2021-01-17 r79842) > * using platform: x86_64-apple-darwin17.0 (64-bit) > * using session charset: UTF-8 > * using options '--no-codoc --no-examples --no-manual --ignore-vignettes --no-stop-on-test-error' > * checking for file 'DelayedMatrixStats/DESCRIPTION' ... OK > * checking extension type ... Package > * this is package 'DelayedMatrixStats' version '1.13.4' > * package encoding: UTF-8 > * checking package namespace information ... OK > * checking package dependencies ... OK > * checking if this is a source package ... OK > * checking if there is a namespace ... OK > * checking for executable files ... OK > * checking for hidden files and directories ... OK > * checking for portable file names ... OK > * checking for sufficient/correct file permissions ... OK > * checking whether package 'DelayedMatrixStats' can be installed ... OK > * checking installed package size ... OK > * checking package directory ... OK > * checking DESCRIPTION meta-information ... OK > * checking top-level files ... OK > * checking for left-over files ... OK > * checking index information ... OK > * checking package subdirectories ... WARNING > Subdirectory 'src' contains no source files. > * checking R files for non-ASCII characters ... OK > * checking R files for syntax errors ... OK > * checking whether the package can be loaded ... OK > * checking whether the package can be loaded with stated dependencies ... OK > * checking whether the package can be unloaded cleanly ... OK > * checking whether the namespace can be loaded with stated dependencies ... OK > * checking whether the namespace can be unloaded cleanly ... OK > * checking dependencies in R code ... NOTE > Missing object imported by a ':::' call: 'DelayedArray:::.reduce_array_dimensions' > Unexported objects imported by ':::' calls: > 'DelayedArray:::.get_ans_type' 'DelayedArray:::RleArraySeed' > 'DelayedArray:::get_Nindex_lengths' 'DelayedArray:::set_dim' > 'DelayedArray:::set_dimnames' 'DelayedArray:::subset_by_Nindex' > 'DelayedArray:::to_linear_index' > See the note in ?`:::` about the use of this operator. > * checking S3 generic/method consistency ... OK > * checking replacement functions ... OK > * checking foreign function calls ... OK > * checking R code for possible problems ... OK > * checking Rd files ... OK > * checking Rd metadata ... OK > * checking Rd cross-references ... OK > * checking for missing documentation entries ... OK > * checking for code/documentation mismatches ... SKIPPED > * checking Rd \usage sections ... OK > * checking Rd contents ... OK > * checking for unstated dependencies in examples ... OK > * checking line endings in C/C++/Fortran sources/headers ... OK > * checking compiled code ... OK > * checking installed files from 'inst/doc' ... OK > * checking files in 'vignettes' ... SKIPPED > * checking examples ... SKIPPED > * DONE > > Status: 1 WARNING, 1 NOTE > See > '/Users/Peter/GitHub/DelayedMatrixStats.Rcheck/00check.log' > for details. >

Aaron Lun (01:42:35): > where was I?

Aaron Lun (01:42:51): > Right, it’s the fact that sourcing the file doesn’t run the test, which is just a little annoying.

Aaron Lun (01:44:26): > If RUnit had the capability to do some like testthat’stest_that({ test expressions here })without their namespace mangling, that would be perfect.

Hervé Pagès (02:37:54) (in thread): > Do you have atestthat.Rfile in thelongtests/folder? Sorry I need to go.

Peter Hickey (19:05:26) (in thread): > > $ cat longtests/testthat.R > library(testthat) > library(DelayedMatrixStats) > > setAutoBlockSize(8) > test_check("DelayedMatrixStats") >

2021-01-30

Aaron Lun (22:16:50): > damn, it’s beautiful. Look forward to the next book rebuild.

2021-01-31

Aaron Lun (15:58:42): > Is there something wrong with the propagation of packages to the landing pages? I’ve been waiting for SCE 1.13.6 to propagate for ages but the website is still stuck on 1.13.4.

Lori Shepherd (16:03:11): > I’ll look tomorrow

2021-02-01

Lori Shepherd (08:23:26): > It looks like 1.13.6 only propagated yesterdayhttp://bioconductor.org/checkResults/devel/bioc-LATEST/SingleCellExperiment/I’ll check the timing of the landing page generation vs the propagation but it may just need more time – I’ll try to adjust if necessary

Lori Shepherd (08:26:42): > Looks like the VIEWS file that generates the landing pages wasn’t updated – we’ll investigate@Hervé Pagès

2021-02-02

Hervé Pagès (03:58:20): > oops, the propagation script was running before the builds were finished. I’ve made some adjustments.:crossed_fingers:

Shubham Gupta (17:39:30): > I am developing a BioConductor package: > In my package, I have includedRemotes: omegahat/Rcompressionin the DESCRIPTION file. But I get installation error in 3.13 build reports:https://bioconductor.org/checkResults/3.13/bioc-LATEST/DIAlignR/malbec2-install.htmlsayingERROR: dependency 'Rcompression' is not available for package 'DIAlignR'This is thelinkfor corresponding commit. What could be the reason for this failure?

Lori Shepherd (17:42:21): > Bioconductor does not allow the use of Remotes. All dependencies must be on CRAN or Bioconductor.

Shubham Gupta (17:44:53): > Thanks. Is there a way to get around it then?

Vince Carey (19:54:18): > @Shubham GuptaRcompression in github is showing 9 years since last modification. Is there a specific compression process that you can make use of only through that package?

2021-02-03

Shubham Gupta (05:36:34): > @Vince CareyI figured out that I can usememDecompressinstead of Rcompression. So this package is not needed anymore

Hervé Pagès (16:23:11): > <!channel>I messed up the builds on malbec2 when I applied system updates and rebooted the machine today so we won’t have a new report tomorrow. Sorry for the inconvenience.

2021-02-05

Hervé Pagès (02:11:43) (in thread): > Sorry for letting this slip my radar. Were you able to sort this out? Works for me. Here is my setup: > > hpages@spectre:~/BiocGenerics/longtests$ ls -l > total 4 > lrwxrwxrwx 1 hpages hpages 17 Feb 4 22:56 testthat -> ../tests/testthat > -rw-rw-r-- 1 hpages hpages 69 Feb 4 22:56 testthat.R > hpages@spectre:~/BiocGenerics/longtests$ cat testthat.R > library(testthat) > library(BiocGenerics) > > test_check("BiocGenerics") > > Then I do: > > hpages@spectre:~/BiocGenerics/longtests$ cd ../.. > hpages@spectre:~$ R CMD build BiocGenerics > * checking for file 'BiocGenerics/DESCRIPTION' ... OK > * preparing 'BiocGenerics': > * checking DESCRIPTION meta-information ... OK > * checking for LF line-endings in source and make files and shell scripts > * checking for empty or unneeded directories > * building 'BiocGenerics_0.37.1.tar.gz' > > and: > > hpages@spectre:~/$ R CMD check --test-dir=longtests --no-stop-on-test-error --no-codoc --no-examples --no-manual --ignore-vignettes --check-subdirs=no BiocGenerics_0.37.1.tar.gz > ... > * checking examples ... SKIPPED > * checking for unstated dependencies in 'longtests' ... WARNING > 'library' or 'require' call not declared from: 'testthat' > * checking tests in 'longtests' ... > Running 'testthat.R' > ERROR > Running the tests in 'longtests/testthat.R' failed. > Last 13 lines of output: > duplicated, eval, evalq, get, grep, grepl, intersect, is.unsorted, > lapply, mapply, match, mget, order, paste, pmax, pmax.int, pmin, > pmin.int, rank, rbind, rownames, sapply, setdiff, sort, table, > tapply, union, unique, unsplit, which.max, which.min > > > > > test_check("BiocGenerics") > ══ Failed tests ════════════════════════════════════════════════════════════════ > ── Failure (test_stuff.R:2:5): some stuff ────────────────────────────────────── > TRUE not identical to FALSE. > 1 element mismatch > > [ FAIL 1 | WARN 0 | SKIP 0 | PASS 0 ] > Error: Test failures > Execution halted > * DONE > > Status: 1 ERROR, 2 WARNINGs, 4 NOTEs > See > '/home/hpages/BiocGenerics.Rcheck/00check.log' > for details. >

Peter Hickey (02:48:04) (in thread): > hmm not on my macbook. I getRemoved empty directory 'DelayedMatrixStats/longtests'duringR CMD buildand consequentlyWARNING directory 'longtests' not foundinR CMD check ...

Peter Hickey (02:48:04) (in thread): > > (base) Peter@Peters-MBP-2 DelayedMatrixStats (master)*$ cd longtests/ > (base) Peter@Peters-MBP-2 longtests (master)*$ ls -l > total 8 > lrwxr-xr-x 1 Peter staff 17 29 Jan 17:02 testthat -> ../tests/testthat > -rw-r--r-- 1 Peter staff 100 3 Feb 18:09 testthat.R > (base) Peter@Peters-MBP-2 longtests (master)*$ cat testthat.R > library(testthat) > library(DelayedMatrixStats) > > setAutoBlockSize(8) > test_check("DelayedMatrixStats") > (base) Peter@Peters-MBP-2 longtests (master)*$ cd ../.. > (base) Peter@Peters-MBP-2 GitHub $ R CMD build DelayedMatrixStats > * checking for file 'DelayedMatrixStats/DESCRIPTION' ... OK > * preparing 'DelayedMatrixStats': > * checking DESCRIPTION meta-information ... OK > * cleaning src > * installing the package to build vignettes > * creating vignettes ... OK > * cleaning src > * checking for LF line-endings in source and make files and shell scripts > * checking for empty or unneeded directories > Removed empty directory 'DelayedMatrixStats/longtests' > Removed empty directory 'DelayedMatrixStats/src' > Removed empty directory 'DelayedMatrixStats/tests/testthat/_snaps' > * building 'DelayedMatrixStats_1.13.5.tar.gz' > > (base) Peter@Peters-MBP-2 GitHub $ R CMD check --test-dir=longtests --no-stop-on-test-error --no-codoc --no-examples --no-manual --ignore-vignettes --check-subdirs=no DelayedMatrixStats_ > DelayedMatrixStats_1.13.4.tar.gz DelayedMatrixStats_c++/ > DelayedMatrixStats_1.13.5.tar.gz > (base) Peter@Peters-MBP-2 GitHub $ R CMD check --test-dir=longtests --no-stop-on-test-error --no-codoc --no-examples --no-manual --ignore-vignettes --check-subdirs=no DelayedMatrixStats_1.13.5.tar.gz > * using log directory '/Users/Peter/GitHub/DelayedMatrixStats.Rcheck' > * using R Under development (unstable) (2021-01-17 r79842) > * using platform: x86_64-apple-darwin17.0 (64-bit) > * using session charset: UTF-8 > * using options '--no-codoc --no-examples --no-manual --ignore-vignettes --no-stop-on-test-error' > * checking for file 'DelayedMatrixStats/DESCRIPTION' ... OK > * checking extension type ... Package > * this is package 'DelayedMatrixStats' version '1.13.5' > * package encoding: UTF-8 > * checking package namespace information ... OK > * checking package dependencies ... OK > * checking if this is a source package ... OK > * checking if there is a namespace ... OK > * checking for executable files ... OK > * checking for hidden files and directories ... OK > * checking for portable file names ... OK > * checking for sufficient/correct file permissions ... OK > * checking whether package 'DelayedMatrixStats' can be installed ... OK > * checking installed package size ... OK > * checking package directory ... OK > * checking DESCRIPTION meta-information ... OK > * checking top-level files ... OK > * checking for left-over files ... OK > * checking index information ... OK > * checking package subdirectories ... OK > * checking R files for non-ASCII characters ... OK > * checking R files for syntax errors ... OK > * checking whether the package can be loaded ... OK > * checking whether the package can be loaded with stated dependencies ... OK > * checking whether the package can be unloaded cleanly ... OK > * checking whether the namespace can be loaded with stated dependencies ... OK > * checking whether the namespace can be unloaded cleanly ... OK > * checking dependencies in R code ... NOTE > Missing object imported by a ':::' call: 'DelayedArray:::.reduce_array_dimensions' > Unexported objects imported by ':::' calls: > 'DelayedArray:::.get_ans_type' 'DelayedArray:::RleArraySeed' > 'DelayedArray:::get_Nindex_lengths' 'DelayedArray:::set_dim' > 'DelayedArray:::set_dimnames' 'DelayedArray:::subset_by_Nindex' > 'DelayedArray:::to_linear_index' > See the note in ?`:::` about the use of this operator. > * checking S3 generic/method consistency ... OK > * checking replacement functions ... OK > * checking foreign function calls ... OK > * checking R code for possible problems ... OK > * checking Rd files ... OK > * checking Rd metadata ... OK > * checking Rd cross-references ... OK > * checking for missing documentation entries ... OK > * checking for code/documentation mismatches ... SKIPPED > * checking Rd \usage sections ... OK > * checking Rd contents ... OK > * checking for unstated dependencies in examples ... OK > * checking installed files from 'inst/doc' ... OK > * checking files in 'vignettes' ... SKIPPED > * checking examples ... SKIPPED > WARNING > directory 'longtests' not found > * DONE > > Status: 1 WARNING, 1 NOTE > See > '/Users/Peter/GitHub/DelayedMatrixStats.Rcheck/00check.log' > for details. >

Peter Hickey (02:56:02) (in thread): > trying again withR CMD build --keep-empty-dirs DelayedMatrixStats…

Peter Hickey (03:05:44) (in thread): > i might be going a bit mad …longtestsis definitely there and non-empty > > (base) Peter@Peters-MBP-2 Desktop $ ls -ls DelayedMatrixStats/ > total 136 > 8 -rw-r--r-- 1 Peter staff 1414 5 Feb 19:01 DESCRIPTION > 8 -rw-r--r--@ 1 Peter staff 332 5 Feb 19:01 DelayedMatrixStats.Rproj > 8 -rw-r--r-- 1 Peter staff 42 5 Feb 19:01 LICENSE > 8 -rw-r--r-- 1 Peter staff 2973 5 Feb 19:01 NAMESPACE > 0 drwxr-xr-x 74 Peter staff 2368 5 Feb 19:01 R > 16 -rw-r--r-- 1 Peter staff 6857 5 Feb 19:01 README.Rmd > 80 -rw-r--r-- 1 Peter staff 37422 5 Feb 19:01 README.md > 8 -rw-r--r-- 1 Peter staff 15 5 Feb 19:01 codecov.yml > 0 drwxr-xr-x 3 Peter staff 96 5 Feb 19:01 inst > 0 drwxr-xr-x 4 Peter staff 128 5 Feb 19:06 longtests > 0 drwxr-xr-x 30 Peter staff 960 5 Feb 19:01 man > 0 drwxr-xr-x 12 Peter staff 384 5 Feb 19:01 man-roxygen > 0 drwxr-xr-x 5 Peter staff 160 5 Feb 19:01 tests > 0 drwxr-xr-x 3 Peter staff 96 5 Feb 19:01 vignettes > > (base) Peter@Peters-MBP-2 Desktop $ ls -ls DelayedMatrixStats/longtests/ > total 8 > 0 lrwxr-xr-x 1 Peter staff 17 5 Feb 19:06 testthat -> ../tests/testthat > 8 -rw-r--r-- 1 Peter staff 100 5 Feb 19:01 testthat.R > > butR CMD buildsays it’s empty > > (base) Peter@Peters-MBP-2 Desktop $ R CMD build --keep-empty-dirs DelayedMatrixStats > * checking for file 'DelayedMatrixStats/DESCRIPTION' ... OK > * preparing 'DelayedMatrixStats': > * checking DESCRIPTION meta-information ... OK > * installing the package to build vignettes > * creating vignettes ... OK > * checking for LF line-endings in source and make files and shell scripts > * checking for empty or unneeded directories > WARNING: directory 'DelayedMatrixStats/longtests' is empty > * building 'DelayedMatrixStats_1.13.5.tar.gz' >

Peter Hickey (03:20:29) (in thread): > fwiw i’ve pushedhttps://github.com/PeteHaitch/DelayedMatrixStats/commit/1de3d1df3b6b290cddb773814c76adbae8fc8d01to BioC git to see if it happens to work on the BioC build machines over the weekend

Dario Righelli (10:20:38): > @Dario Righelli has joined the channel

Hervé Pagès (12:49:12) (in thread): > Gosh, I get the same thing with the latest DelayedMatrixStats: > > hpages@spectre:~$ bdevbuild --no-build-vignettes --keep-empty-dirs DelayedMatrixStats > * checking for file 'DelayedMatrixStats/DESCRIPTION' ... OK > * preparing 'DelayedMatrixStats': > * checking DESCRIPTION meta-information ... OK > * checking for LF line-endings in source and make files and shell scripts > * checking for empty or unneeded directories > WARNING: directory 'DelayedMatrixStats/longtests' is empty > * building 'DelayedMatrixStats_1.13.5.tar.gz' > > We must both be crazy to believe that this directory is not empty! It doesn’t matter how much stuff I put in there,R CMD buildalways issues the silly warning and removes all the content of the folder when creating the tarball. Some more investigation will be needed coz’ right now we’ll probably see the same thing happen on the build machines.

Hervé Pagès (13:04:35) (in thread): > IT’S YOUR .RBUILDIGNORE FILE!!! Hmm, sorry, let me try this again. You know what, I think it might be something to do with your .Rbuildignore file, maybe, probably, or something like that.

Peter Hickey (14:27:57) (in thread): > bloody hell. i thought i looked at that to check.

Peter Hickey (14:28:45) (in thread): > and i’ve no idea why i added it to.Rbuildignorein the first place

Hervé Pagès (14:30:41) (in thread): > The thing is, I didn’t see it in your.Rbuildignore. I just removed the file entirely and thenR CMD buildstopped complaining about thelongtestsfolder being empty.

Peter Hickey (14:31:27) (in thread): > anyway, i removed it from.Rbuildignoreand it now works as expected. Thanks for your help!

Peter Hickey (14:32:24) (in thread): > aside after removing it revealed some weird bug in the vignette > > (base) Peter@Peters-MBP-2 GitHub $ R CMD build DelayedMatrixStats > * checking for file 'DelayedMatrixStats/DESCRIPTION' ... OK > * preparing 'DelayedMatrixStats': > * checking DESCRIPTION meta-information ... OK > * installing the package to build vignettes > * creating vignettes ... ERROR > --- re-building 'DelayedMatrixStatsOverview.Rmd' using rmarkdown > Warning in engine$weave(file, quiet = quiet, encoding = enc) : > Pandoc (>= 1.12.3) and/or pandoc-citeproc not available. Falling back to R Markdown v1. > Quitting from lines 19-44 (DelayedMatrixStatsOverview.Rmd) > Error: processing vignette 'DelayedMatrixStatsOverview.Rmd' failed with diagnostics: > could not find function "Biocpkg" > --- failed re-building 'DelayedMatrixStatsOverview.Rmd' > > SUMMARY: processing the following file failed: > 'DelayedMatrixStatsOverview.Rmd' > > Error: Vignette re-building failed. > Execution halted >

Peter Hickey (14:32:36) (in thread): > which i fixed by addign alibrary(BiocStyle)

Peter Hickey (14:32:57) (in thread): > but don’t know why it only became visible after the change to.Rbuildignore

Hervé Pagès (14:33:52) (in thread): > your.Rbuildignorefiles are powerful!

2021-02-06

Aaron Lun (03:19:04): > Hm. Many basilisk clients are timing out on machv2.

Aaron Lun (03:19:15): > But not all of them, oddly enough.

Aaron Lun (03:20:19): > Actually, it’s just velociraptor and snifter. Everyone else is fine.

Aaron Lun (03:20:48): > Hm. I wonder why that is.@Kevin Rue-Albrecht@Alan O’Ccan you bump your versions? Maybe there’s something wrong with the cached environments.

2021-02-07

Alan O’C (18:04:20): > Timed out again. I’ll see if I can pull up any hints tomorrow

2021-02-08

Aaron Lun (02:51:27): > @Hervé Pagèsall books should now be dumping their HTMLs ininst/doc/book.

Aaron Lun (22:06:36): > @Alan O’CI notice there are frequent bursts of multi-core activity when Itopduring a snifter vignette. Maybe something to do with the somewhat-recently added OMP thread limit on the build machines?

Aaron Lun (22:10:06): > I can repro aconsiderableslowdown when I setexport OMP_THREAD_LIMIT=2before running the vignette.

Aaron Lun (22:10:26): > I don’t really understand why this is, either, because it wasn’t using a lot of threads in the first place…

Aaron Lun (22:11:14): > 5 minutes and counting, for something that normally runs in <1 minute.

Aaron Lun (22:13:08): > 9 minutes. I think tis is the culpirt.

2021-02-09

Alan O’C (07:41:11): > Makes sense, cheers

Alan O’C (07:42:22): > Well… in a very limited way it makes sense, given that njobs is set to 1

Alan O’C (11:56:55): > SettingOMP_THREAD_LIMIT=2in github actions reproduces it. Now for the fun of remote debugging a different OS

Dan Bunis (13:09:09): > Hadn’t actually noticed this before but is the Release branch testing on a non-daily schedule? Not sure if I’m just not looking in the right spot versus it not being on the website anywhere… Would I be correct in thinking Release versions get built/tested on a similar schedule as when stats are updated — so Tues / Fri?

Lori Shepherd (13:13:45): > Release follows the same schedule as devel. The software should be daily, This page indicates when other types are runhttp://bioconductor.org/checkResults/

Lori Shepherd (13:14:14): > however, the Release daily software build appears to be broken and we are currently investigating

Dan Bunis (13:17:34): > ahhh, okay, thanks Lori. I was just confused cuz of the current bug then.

Hervé Pagès (16:10:29): > An update on the status of the release builds: Mac builder merida1 is down which is why we didn’t have a new report since last Friday. We should get a new report tomorrow but without merida1 in it. What happened to merida1 is currently under investigation.

Alan O’C (18:45:23): > Is there a policy on setting env variables within a package? Asking because I’m getting timeout whenOMP_THREAD_LIMITis set butOMP_NUM_THREADSis not, and only on MacOS somehow. Though I guess that might point to something that needs patched in the BBS settings

Hervé Pagès (21:48:04): > I don’t think packages should set their own env variables. That would kind of defeat the purpose of env variables. > > So in an interactive session on machv2 (Mac builder) and withOMP_THREAD_LIMIT=2, the snifter vignette gets stuck at: > > > fit <- fitsne(mat, random_state = 42L) > OMP: Warning #96: Cannot form a team with 3 threads, using 2 instead. > OMP: Hint Consider unsetting KMP_DEVICE_THREAD_LIMIT (KMP_ALL_THREADS), KMP_TEAMS_THREAD_LIMIT, and OMP_THREAD_LIMIT (if any are set). > > Hmm so why would Python function openTSNE.TSNE() (the workhorse behindfitsne()) try to “form a team with 3 threads” when the default value forn_jobsis 1? Also why using only 2 threads instead of 3 would cause that thing to hang forever? Finally the icing on the cake: sadly trying to interrupt this with CTRL+C doesn’t seem to work.@Aaron LunDo you think this is a general problem of keyboard interruptions not propagating to the Python code thru the basilisk/reticulate interface? Or is it just the low-level C code in openTSNE.TSNE() that doesn’t handle interruptions?@Alan O’CEarlier you said that you were able to reproduce this timeout in GitHub Actions. Doesn’t this point to a real problem in how openTSNE.TSNE interacts with OpenMP on Mac? If you say that the timeout goes away by havingOMP_THREAD_LIMITandOMP_NUM_THREADSbothset (e.g. to 2) then we could try this on the builders even though that seems like randomly shooting in the dark until we hit a setup that makes openTSNE.TSNE() happy on the builders. The problem would remain for Mac users that haveOMP_THREAD_LIMITset to 2.

Aaron Lun (21:49:12): > Whatever the control-C problem is, it’s deeper than ****basilisk****. Possibly ****reticulate****, but most probably at whatever openTSNE is calling. Not easily fixable at my level, I’m afraid. Kind of like C++ code withotu interrupt handling.

Alan O’C (21:49:24): > openTSNE ran on its own in a test project with justOMP_THREAD_LIMITand didn’t hang. It also complains about not being able to form a team with 3 threads on linux but runs fine

Hervé Pagès (21:52:26): > but it hangs in a reproducible manner when run thru snifter with OMP_THREAD_LIMIT set to 2 on Mac. Sounds like this would deserve some investigation.

Aaron Lun (21:53:35): > @Alan O’Cwhat happens with raw reticulate, no basilisk?

Aaron Lun (21:53:46): > Don’t activate the environment either.

Hervé Pagès (22:11:03): > I can confirm that setting bothOMP_THREAD_LIMITandOMP_NUM_THREADSto 2 solves the problem. The builds will use these settings from now on.

Alan O’C (22:12:04): > I’ll do my best to narrow the problem down and get it fixed upstream as well

2021-02-10

Alan O’C (08:12:58) (in thread): > python only, reticulate only, and basilisk only all seem to workhttps://github.com/Alanocallaghan/testing-openmp-macos

Aaron Lun (12:06:39) (in thread): > with and withotu the extra envvar?

Alan O’C (12:07:15) (in thread): > Just withOMP_THREAD_LIMIT, withoutOMP_NUM_THREADS

Aaron Lun (12:08:51) (in thread): > hm. so it just doesn’t work inside the vigbnette?

Alan O’C (13:51:55) (in thread): > Works in a normal R session so seems like it might be that. Or could be within R CMD build in general I suppose

Aaron Lun (13:53:01) (in thread): > because I was just runningrmarkdown::renderon the vignette.

Aaron Lun (13:53:10) (in thread): > not even R CMD build.

Alan O’C (16:27:42) (in thread): > It depends on the dimensions of the data

Alan O’C (17:57:35) (in thread): > Yeah I lied, it does happen with plain basilisk but not with reticulate, nor in python. The env variable is passed through at least according to printingSys.getenvwithin thebasiliskRuncall

Aaron Lun (18:12:00) (in thread): > Hmm… maybe something to do with the activation of the conda environment?

Alan O’C (18:18:12) (in thread): > Yeah conda might be clobbering the env variable, I’ll check

Alan O’C (18:18:28) (in thread): > For posterity this will reproduce on MacOS: > > library("basilisk") > > python_env <- BasiliskEnvironment( > "fitsne", > pkgname = "snifter", > packages = c( > "opentsne=0.4.3", > "scikit-learn=0.23.1", > if (basilisk.utils::isWindows()) "scipy=1.5.0" else "scipy=1.5.1", > "numpy=1.19.0" > ) > ) > basiliskRun( > fun = function(...) { > t <- reticulate::import("openTSNE") > nrow <- 3000 > ncol <- 20 > print(Sys.getenv("OMP_THREAD_LIMIT")) > mat <- matrix(rnorm(nrow * ncol), nrow = nrow, ncol = ncol) > t$TSNE()$fit(mat) > }, > env=python_env > ) >

Alan O’C (18:18:55) (in thread): > Obviously with a suitable.Renviron

Aaron Lun (18:20:23) (in thread): > the activation should occur before OMP_THREAD_LIMIT is checked, so if it’s still there at that point, it might be another problme.

Alan O’C (18:32:40) (in thread): > Printing it in the anonymous function returned “2” which is presumably after activation

2021-02-12

Jianhai Zhang (21:35:20): > Hi, I have an error athttp://bioconductor.org/checkResults/devel/bioc-LATEST/spatialHeatmap/tokay2-install.html.

Jianhai Zhang (21:36:11): > Is it because the windows system lack HDF5Array?

Jianhai Zhang (21:36:28): > > ERROR: dependency 'HDF5Array' is not available for package 'spatialHeatmap' >

2021-02-13

Hervé Pagès (15:01:19) (in thread): > Thx. The new settings (OMP_THREAD_LIMIT = OMP_NUM_THREADS = 2) solved the problem for snifter but not for velociraptor: > * https://bioconductor.org/checkResults/3.13/bioc-LATEST/snifter/machv2-buildsrc.html > * https://bioconductor.org/checkResults/3.13/bioc-LATEST/velociraptor/machv2-buildsrc.html > @Kevin Rue-Albrecht@Aaron LunAny idea what’s going on with velociraptor? Can’t do this at the moment but I’ll try to go on machv2 and do some troubleshooting in the next couple of days.

Aaron Lun (15:03:47) (in thread): > Hm… will boot up my mac later to try to figure this out.

Aaron Lun (15:12:57) (in thread): > The funny thing is that it’s also happening in release, so it can’t be a change to the basilisk code. Plus the conda versions are all pinned.

Hervé Pagès (15:25:31): > @Jianhai ZhangI added support for LZF compression recently to HDF5Array but it seems that the Rtools40 64-bit compiler doesn’t like the LZF code. I just committed a fix (it’s in HDF5Array 1.19.4). Let’s see how it goes in the next couple of days.

Jianhai Zhang (19:15:50): > Thanks!

Aaron Lun (19:26:21) (in thread): > @Hervé Pagèswas able to reproduce the timeout, and apparently the fix is: > > export KMP_DEVICE_THREAD_LIMIT=2 > export KMP_TEAMS_THREAD_LIMIT=2 > > I have ABSOLUTELY NO IDEA why this works.

Aaron Lun (19:30:43) (in thread): > intel, huh.

Alan O’C (19:34:02) (in thread): > What’s KMP here? Just a different set of OpenMP flags?

Aaron Lun (19:35:32) (in thread): > intel’s version of OpenMP, apparently. Really hard to find docs on this.

Aaron Lun (19:41:16) (in thread): > More precisely, I believe these are Intel’s extensions to OpenMP, to be used with the intel compiler suite.

Alan O’C (19:51:05) (in thread): > Ah yeah, I could only dig up error messages relating to KMP but I assumed it was equivalent the libgomp extensions

Hervé Pagès (22:23:29) (in thread): > Thx@Aaron Lun, I guess I’ll add these settings to the builds. Hopefully at some point someone manages to understand what’s really going on here and we stop shooting in the dark.

Aaron Lun (23:11:09) (in thread): > amen to that

Hervé Pagès (23:38:10) (in thread): > Done:https://github.com/Bioconductor/BBS/blob/32afb38b7c546976425692b38ff168c21017e378/3.13/Renviron.bioc#L46-L48

Aaron Lun (23:54:54) (in thread): > :crossed_fingers:

2021-02-14

Aaron Lun (21:42:20): > Ugh, this “directory expiration based on access time” is a real PITA when considering thread safety.

2021-02-15

Hervé Pagès (16:30:33) (in thread): > All seems good now as far as compilation on Windows goes soHDF5Arrayat least got installed on tokay2. There’s still the issue that the examples time out on this machine, which prevents the new package binary to propagate, but that shouldn’t affect the build results forspatialHeatmapand other packages that depend directly or indirectly onHDF5Array.

Jianhai Zhang (16:57:18) (in thread): > Sure, thank you!

2021-02-16

Aaron Lun (23:36:27): > @Hervé PagèsI recently rejigged the subbooks so that some of them (OSCA.basic,OSCA.advanced) listOSCA.workflowsas a dependency. I was expecting that the build system would respect this dependency, building and installingOSCA.workflowsfirst before building and installingOSCA.basicandOSCA.advanced. The general idea is to allowOSCA.workflowsto populate a cache of serialized objects that can be re-used in the other books, enabling seamless cross-book communication within the BBS. > > If it works, it would be pretty cool; no redundant computation when two books need to do the same thing, which should take a lot of burden off the build system. However, it seems that all these jobs are starting at the same time (2021-02-16 10:31:10), which suggests that they’re not being run in the expected order. Is this because the build system doesn’t expect books to depend on other books?

2021-02-17

Hervé Pagès (02:13:06): > The build system installs all the packages to build and check first (and when it does so it of course walks the DAG of deps in the right direction). This is the INSTALL step (left-most column on the build report). Then it performs the BUILD step i.e. it runsR CMD buildon all the packages. It does this in any order. Maybe it’s alphabetical order, can’t remember. But it should not matter. In particular, if package B depends on package A, once the two packages are installed, you should be able to runR CMD buildon B first and then on A right?

Hervé Pagès (02:14:15): > Or run the two commands at the same time (something that the build system does a lot).

Aaron Lun (02:27:17): > Hm. I see.

Aaron Lun (02:28:36): > I misunderstood the order of events here.

Aaron Lun (02:29:30): > Well. Bum. I guess I could shift the creation of the objects into the install, but that would causeinstallto take a long time. Plus side is that thebuildwould be super-fast because it would just re-use the objects that it generated previously.

Hervé Pagès (02:37:59): > Sounds good as long as the objects end up inuser_data_dir()or something like that and not in the package installation directory, which I suppose is already the case?

Aaron Lun (02:38:22): > yes, that’s the plan.

Aaron Lun (02:42:04): > Come to think of it… I might as well just build the book during installandstow away the compiled HTMLs ininst/doc/bookat the same time. Thenbuildwould literally do nothing. And it would be as easy as moving all the instructions fromvignettes/Makefiletoconfigure.

Hervé Pagès (02:57:08): > hmn… does this mean that when people try to install the book packages withBiocManager::install()it’ll take 5 hours?

Aaron Lun (02:59:27): > ah shit.

Aaron Lun (02:59:33): > I forgot we wanted to allow people to do that.

Aaron Lun (03:03:52): > Well, this is quite the pickle. Will need to sleep on it. I could impose file locks on the external directory, but it would defeat the purpose of the parallel builds.

Aaron Lun (03:15:24): > With some very careful file locking, this can be done without forcing long wait times for serial processing.

2021-02-18

Aaron Lun (02:11:00): > It’s done, but I’m not entirely confident that it works. I was getting some kind of race problem but then it went away and I don’t know why.

Hervé Pagès (03:22:24): > Let’s see what happens on Friday.

Aaron Lun (04:34:55): > aw man. Figured it out.

Aaron Lun (04:37:16): > Overeager deletion of expired caches. Should be solved byhttps://github.com/Bioconductor/Contributions/issues/1889.

2021-02-19

Aaron Lun (22:54:24): > well, the good news is that the build failures aren’t incomprehensible.

2021-02-20

Aaron Lun (01:07:57): > I’ve got to say, though. These books are great as high-level integration tests.

2021-02-26

Lori Shepherd (11:43:55): > The software release build report has not been updated since monday. we have identified the issue and are working on a resolution. We appreciate your patience while the issue is being resolved.

2021-02-27

Aaron Lun (01:29:47): > Slowly crawling towards some successful book builds…

Aaron Lun (03:34:24): > @Hervé PagèsI may need to ask you to delete the~/.cache/rebookdirectory on rex3. The latest book build failures don’t make any sense and I’m trying to figure out whether they’re due to a stale cache.

Aaron Lun (03:35:02): > If you could find a small interval to fire off a build, just to see whether they get past the point of the existing errors, that would also be helpful.

Hervé Pagès (15:11:14): > Just deleted~/.cache/rebookand restarted the book builds on rex3. Of the 3 failing subbooks, the 1st one to reach the point of existing error should beOSCA.basicin about 30 min. I’ll let you know.

Aaron Lun (15:12:14): > thanks

Hervé Pagès (15:22:20): > well, of course it might take longer because the cache needs to be repopulated: > > biocbuild@rex3:~/.cache/rebook$ du -sh * > 820K OSCA > 305M OSCA.advanced > 1.2G OSCA.basic > 685M OSCA.intro > 5.3G OSCA.workflows > > so far, so good

Aaron Lun (15:29:04): > yup

Aaron Lun (15:34:26): > AHA. I finally managed to repro the basic error on my laptop.

Hervé Pagès (15:42:11): > Still populating the cache: > > biocbuild@rex3:~/.cache/rebook$ du -sh * > 820K OSCA > 834M OSCA.advanced > 1.2G OSCA.basic > 685M OSCA.intro > 8.0G OSCA.workflows > > Unfortunately this is going to take much longer than what I thought and the software builds on rex3 just started, which might create some conflicts (re-installation of the software packages might break the book builds).

Hervé Pagès (15:43:33): > But if we are only interested to what happens toOSCA.basic, we might still be good…

Aaron Lun (15:49:01): > that’s okay, let’s see how it goes. I’m confident that the OSCA.advanced and OSCA.workflows errors should be resolved. But the OSCA.basic error is WEIRD.

Aaron Lun (15:49:56): > It seems like a race condition with BiocFileCache…

Hervé Pagès (15:50:02): > OSCA.basic:Failed again, with the same error: > > # label: corral-sort (with options) > # List of 3 > # $ fig.width : num 10 > # $ fig.height: num 6 > # $ fig.cap : chr "Dimensionality reduction results of all pool-and-split libraries in the SORT-seq CellBench data, computed by a "| *_truncated_* > # > # Loading required package: dbplyr > # Quitting from lines 389-413 (reduced-dimensions.Rmd) > # > # Error in lazyLoadDBinsertValue(data, datafile, ascii, compress, envhook) : > # long vectors not supported yet: ../../../rdownloads/R-4.1.r79979/src/main/connections.c:6015 > > Error in compileChapter(path) : > failed to compile '~/.cache/rebook/OSCA.basic/0.99.4/reduced-dimensions.Rmd' > Calls: <Anonymous> -> .locked_compile_chapter -> compileChapter > Execution halted > make: ***** [Makefile:4: compiled] Error 1 > Error in tools::buildVignettes(dir = ".", tangle = TRUE) : > running 'make' failed > Execution halted >

Hervé Pagès (15:52:56): > OSCA.advanced: > > # label: unnamed-chunk-7 > # Quitting from lines 174-183 (droplet-processing.Rmd) > # > # Error in SummarizedExperiment:::.SummarizedExperiment.charbound(subset, : > # index out of bounds: RP11-34P13.7 RP11-34P13.8 ... AC004556.1 AC240274.1 > > Error in compileChapter(path) : > failed to compile '~/.cache/rebook/OSCA.advanced/0.99.4/droplet-processing.Rmd' > Calls: <Anonymous> -> .locked_compile_chapter -> compileChapter > Execution halted > make: ***** [Makefile:4: compiled] Error 1 > Error in tools::buildVignettes(dir = ".", tangle = TRUE) : > running 'make' failed > Execution halted >

Aaron Lun (15:53:28): > Ugh.

Aaron Lun (15:53:38): > Well. Back to the drawing board.

Aaron Lun (15:57:16): > I wonder what this connections thing is. I shall think about it as I walk to lunch.

Aaron Lun (20:20:39): > surreal. this is surreal.

Aaron Lun (20:21:23): > The same code behaves differently when run in R CMD build vs on the console.

Aaron Lun (20:34:08): > getting closer. Apparently ****knitr**** really doesn’t like being run inside another package.

Hervé Pagès (21:12:20) (in thread): > Kind of related to this, I’ve run into situations in the past where things likesort(<character>)orfactor(<character>)would not return the same thing depending on whether they’re run insideR CMD checkor interactively, because IIRCR CMD checkalways resets theLC_COLLATEto"C", or something like that. Or maybe it does it only before running the tests, can’t remember. I’ve learned since to always specify thelevelsargument (e.g.factor(x, levels=unique(x))) when turning a character vector into a factor.

Aaron Lun (22:34:27): > well, I’ve boiled it down to some interaction between the fact that I need toload()an RData file’s contents; ****knitr****’s caching, which creates another RData file; and ****callr****’s creation of a new R process. All these three ingredients are necessary to trigger the error.

Aaron Lun (22:59:08): > Well. The first error isprobablyfixed, though the underlying cause is unknown. I justrm’d theenvand went on with my day.

2021-02-28

FelixErnst (04:28:48) (in thread): > I thought I was the only one…

Aaron Lun (06:11:29): > environemnts, man. Environments.

2021-03-09

Hervé Pagès (13:14:44): > @Aaron LunJust a heads-up that I’m going to replace rex3 with malbec2 for the book builds, and run these builds 3 times a week (Mon, Wed, Fri) instead of 2 (Tue and Thu) . All the 3.13 builds now run on rex3 so malbec2 has a lot of free time.

Aaron Lun (13:14:59): > okey dokey

2021-03-10

Hervé Pagès (12:12:17): > done

2021-03-13

Aaron Lun (23:38:42): > rebook now has a Suggests: OSCA.intro, to demonstrate how to link between books. Are the book packages useable as dependencies from the perspective of the BBS?

2021-03-14

Hervé Pagès (18:40:50): > The non-book builds had no knowledge of the books repo (https://bioconductor.org/packages/3.13/books). I just changed that so they should be able to install the book source tarballs from this repo.

Aaron Lun (20:56:49): > great, thanks

2021-03-15

Hervé Pagès (19:31:42): > @Aaron LunI see old/stale stuff on the Windows builder e.g.: > > PS C:\Users\biocbuild\AppData\Local\rebook\rebook\OSCA.workflows> C:\rtools40\usr\bin\ls -al > total 48 > drwxr-xr-x 1 biocbuild None 0 Mar 15 04:02 . > drwxr-xr-x 1 biocbuild None 0 Feb 16 10:32 .. > -rw-r--r-- 1 biocbuild None 0 Feb 19 10:32 0.99.3-00LOCK > -rw-r--r-- 1 biocbuild None 0 Feb 26 10:32 0.99.4-00LOCK > -rw-r--r-- 1 biocbuild None 0 Mar 2 10:32 0.99.5-00LOCK > -rw-r--r-- 1 biocbuild None 0 Mar 9 10:32 0.99.6-00LOCK > drwxr-xr-x 1 biocbuild None 0 Mar 15 04:56 0.99.7 > -rw-r--r-- 1 biocbuild None 0 Mar 10 12:02 0.99.7-00LOCK > > or: > > PS C:\Users\biocbuild\AppData\Local\rebook\rebook\OSCA.basic> C:\rtools40\usr\bin\ls -al > total 36 > drwxr-xr-x 1 biocbuild None 0 Mar 15 04:02 . > drwxr-xr-x 1 biocbuild None 0 Feb 16 10:32 .. > -rw-r--r-- 1 biocbuild None 0 Feb 19 10:32 0.99.3-00LOCK > -rw-r--r-- 1 biocbuild None 0 Feb 23 10:32 0.99.4-00LOCK > -rw-r--r-- 1 biocbuild None 0 Mar 2 10:32 0.99.5-00LOCK > -rw-r--r-- 1 biocbuild None 0 Mar 5 10:32 0.99.6-00LOCK > -rw-r--r-- 1 biocbuild None 0 Mar 9 10:32 0.99.7-00LOCK > -rw-r--r-- 1 biocbuild None 0 Mar 10 12:02 0.99.8-00LOCK > drwxr-xr-x 1 biocbuild None 0 Mar 15 04:51 0.99.9 > -rw-r--r-- 1 biocbuild None 0 Mar 15 04:02 0.99.9-00LOCK > > Is it ok to clean?

Aaron Lun (19:35:31): > oh, I forgot to delete the LOCK files. Yeah, it should be okay to clean those. I will add some checks to ****dir.expiry**** to clean out unused lock files.

Hervé Pagès (19:49:01): > sounds good

Aaron Lun (19:49:50): > well, assuming it ever makes it past review.

2021-03-16

Aaron Lun (01:38:20): > actually,deletion of those lock files is going to be more troublesome than I thought. Their existence is necessary to ensure thread safety, and deleting them would require creation of another lock file somewhere to make sure that the deletion itself is thread-safe.

Aaron Lun (02:14:44): > well. I think I did it. Not entirely sure.

2021-03-17

Aaron Lun (02:14:58): > are we going to get landing pages for the book packages? This would give them the same kind of visibility as the software/workflow/data packages. FWIW I’ve set up the stub vignettes so that clicking on them will auto-redirect to the book packages.

Hervé Pagès (02:52:46): > > are we going to get landing pages for the book packages? > Eventually, once things settle down. FWIW in the meantime I manually added temporary book index pages a few months ago for release (https://bioconductor.org/books/release/) and devel (https://bioconductor.org/books/devel/). The former is linked fromhttps://bioconductor.org/help/. They’re static:clown_face:

Kozo Nishida (05:08:54): - Attachment: Attachment > Many Bioconductor packages depending on Seurat 4.0.0 was unable to build or install. > https://github.com/satijalab/seurat/issues/4226

Hervé Pagès (11:26:39): > Affected Bioconductor packages: dittoSeq, escape, fcoex, Nebulosa, phemd, pipeComp, progeny, scAlign, scBFA, scCB2, scDataviz, schex, scMAGeCK, scTensor, scTGIF, singleCellTK, Spaniel, TAPseq, tidybulk, zinbwave. All broken in release and devel. Who knew so many BioC packages depended on Seurat?:pensive:

Aaron Lun (11:27:09): > ah well. It could be worse.

Dan Bunis (13:38:29): > Any suggestion of what to do vs wait for the Seurat team to fix themselves? Probably just wait for now? Looking at this list, I think at least some of these may be broken because they rely on dittoSeq. I can push its transition away from Seurat further, but it’s currently intertwined into the vignette & all unit testing scripts. So not a quick fix.

Mike Smith (13:45:01): > It looks like the Seurat guys have already fixed it in their own devel branch, and I guess it’ll make its way onto CRAN in the not too distant future. I’d suggest just waiting for now (that’s easily said when it’s not my package that’s affected!)

Hervé Pagès (13:51:09): > Seehttps://github.com/satijalab/seurat/issues/4226#issuecomment-800625766for a temporary workaround. - Attachment: Comment on #4226 Error: package or namespace load failed for ‘Seurat’: object ‘markvario’ is not exported by ‘namespace:spatstat’ > Hi all, > > We are aware of this issue and working on submitting a fix to CRAN. In the meantime, this issue has been resolved on the develop branch of Seurat. To install the development version of Seurat, please see the instructions https://satijalab.org/seurat/articles/install.html#install-the-development-version-of-seurat-1|here.

Dan Bunis (13:53:53) (in thread): > couple comments above:https://github.com/satijalab/seurat/issues/4226#issuecomment-800019108 - Attachment: Comment on #4226 Error: package or namespace load failed for ‘Seurat’: object ‘markvario’ is not exported by ‘namespace:spatstat’ > The spatstat people answered this: > > “I’m sorry, but this is not a bug. The function has been moved to a sub package and the maintainer of Seurat has been informed and needs to update the package. The error will go away when this is done. Did you report the error to the maintainer?” > > I have copied this answer here to finish up this discussion with as much info as I have. > > Looking forward to the Seurat fix.

Dan Bunis (13:54:15) (in thread): > major eye rolls

Hervé Pagès (13:55:25) (in thread): > The question is: have you tried to install thedevelopbranch of Seurat? Does it fix the problem?

Mike Smith (13:55:26): > On a related note, I recently added adependsOnargument toBiocPkgTools::problemPage(). This gets you an overview of the build status of all packages that depend/import/suggest some other package. As someone whole maintains a few packages with a lot of reverse dependencies, I find it helpful to check this a few days after I’ve commit a change. It’s reassuring if they don’t all turn to “error” and gives me a heads up something unexpected happened if they do. > > If I was a Seurat maintainer, I could doBiocPkgTools::problemPage(authorPattern = "", dependsOn = "Seurat")and get back: - File (PNG): image.png

Dan Bunis (13:56:21): > This is wonderful, thanks@Mike Smith!

Dan Bunis (13:57:35) (in thread): > I haven’t yet but I’m sure it would… The error for dittoSeq in the build reports is exactly as listed by others in the issue.

Dan Bunis (13:57:58) (in thread): > eh… it’s a quick check. I’ll do it now.

Dan Bunis (14:05:33) (in thread): > > remotes::install_github(repo = 'satijalab/seurat', ref = 'develop') > ... > * DONE (Seurat) > Adding 'Seurat_4.0.0.9015.tgz' to the cache > > Restarting R session... > > > library(Seurat) > Attaching SeuratObject > > >

Aaron Lun (14:05:51): > ah, but mike, if you were a seurat maintainer, this wouldn’t have happened in the first place.

Dan Bunis (14:06:59): > Indeed lol. I am going to shrink dittoSeq’s dep on Seurat for sure… this isn’t the first time, and it won’t be the last.

Hervé Pagès (14:07:19): > hmm, I wonder why scTGIF and singleCellTK don’t show up in your list@Mike Smith

Dan Bunis (14:11:43) (in thread): > R CMD Check just completed fine too. (R 4.0 on Mac)

Marcel Ramos Pérez (14:24:02): > Maybe it’s looking at immediate deps? I don’t see Seurat listed in the Imports etc. fieldshttp://www.bioconductor.org/packages/release/bioc/html/scTGIF.html

Mike Smith (14:52:53): > Yep, the dependencies come fromBiocPkgTools::buildPkgDependencyDataFrame()and I think that only looks at direct dependencieshttps://rdrr.io/bioc/BiocPkgTools/man/buildPkgDependencyDataFrame.html

Mike Smith (14:57:01) (in thread): > I think you might be forgetting the time I pushed a breaking change torhdf5a week before release and then went on holiday:see_no_evil:

Hervé Pagès (15:07:50): > singleCellTK imports Seurat though

Mike Smith (16:13:38): > Not sure then. I’ve not looked at the code behind that function. Would be good to know if it’s missing something.

Mike Smith (16:42:24): > Ok, the default behaviour is only to show the build stages that don’t pass. I was then filtering by builder and install results in the search box. The install step for singleCellTK is passing, so it’s not included, but the check step fails and is show: - File (PNG): image.png

Hervé Pagès (16:52:03): > Nice detective work, thanks Mike! I like this kind of summary and the convenience it adds to the development process. Now I want to add something like this to the build report e.g. each package would get a mini build report for all its reverse direct deps on their summary pagehttps://bioconductor.org/checkResults/3.13/bioc-LATEST/rhdf5/(I also want to provide links to the package summary page from the main report).

Hervé Pagès (16:56:59): > Anyways, we’re dealing with a more serious issue at the moment.<!channel>rex3 has been unresponsive for the last 24h or so. We’re still waiting to hear back from the IT folks at BWH to know more about what happened to it exactly. In the mean time, the devel builds are stale. Sorry for the inconvenience.

Charlotte Soneson (17:09:33) (in thread): > I’m wondering whether it would make sense to work in something like what’s been discussed inhttps://kevinrue.github.io/BiocChallenges/articles/challenges/build_failure_tracking.htmlas well - this rather takes the opposite direction and tries to figure out why a certain package fails. - Attachment (kevinrue.github.io): Facilitating diagnosis of Bioconductor package build failures > BiocChallenges

2021-03-18

Hervé Pagès (12:45:05): > The mini reports for direct reverse deps are implemented e.g.https://bioconductor.org/checkResults/3.12/bioc-LATEST/CRISPRseek/orhttps://bioconductor.org/checkResults/3.12/bioc-LATEST/rhdf5/etc…

Mike Smith (12:58:15): > Nice! Will we get them for devel too?

Hervé Pagès (12:58:47): > Yes, as soon as we get rex3 back

Mike Smith (13:03:58): > Would it be possible to get a version of the “Quick Stats” table that we have on the index page, but for each mini report? My use of the tables I was showing above is to get a quick summary of how many reverse dependencies are broken vs how many are fine. The quick stats table does that really nicely.

Hervé Pagès (13:06:01): > Sure. I’ll try to add this.

Hervé Pagès (14:56:15): > Done:https://bioconductor.org/checkResults/3.12/bioc-LATEST/rhdf5/

Hervé Pagès (14:57:54): > (links on the build nodes are broken though)

2021-03-19

koki (01:34:21): > @koki has joined the channel

FelixErnst (08:32:55): > I have an issue with a build on the SPB for package submission.http://bioconductor.org/spb_reports/mia_buildreport_20210319083721.htmlDuring R CMD check an experimental packagemicrobiomeDataSetsis not found. This is specific to tokay2 and does not occur on malbec2 or machv2. Did this type of issue ever come up? The experimental package builds find on riesling1. Can I do anything about it?

Lori Shepherd (09:36:45): > I’ll try and look into why it isn’t available for windows. Thanks for reporting

Lori Shepherd (10:52:37) (in thread): > Not sure why tokay2 wasn’t finding microbiomeDataSets. I have temporarily installed the package so that your SPB issue should find it on next build

FelixErnst (10:53:03) (in thread): > should I bump the version number to trigger the build?

Lori Shepherd (10:53:23) (in thread): > yes that would be good.

Lori Shepherd (10:53:46) (in thread): > let me know if there is still an issue

Pierre-Luc Germain (11:12:27): > Hi folks, I’ve got a build error on Bioc (linux/mac only) which is driving me crazy, I tried many things and can’t get rid of it, and neither I nor anybody I asked could reproduce it in any other environment:http://bioconductor.org/spb_reports/diffUTR_buildreport_20210318132656.htmlIt’s an example callingRsubread::featureCounts, which then crashes with: > > ***** caught segfault ***** > address 0x10, cause 'memory not mapped' > > In case anyone has seen something like this before or has any idea what’s going on, any tip would be greatly appreciated!

FelixErnst (11:24:56) (in thread): > The issue is solved. The remaining warning is due to the known issue of invalid names of intermediate man pages on windows (The names include<). Thanks for your help!

Mike Smith (11:45:42) (in thread): > Memory not mapped is never a nice error to see! It normally means some underlying C code is doing something it shouldn’t. In my experience, if it’s happening on one system it’s almost certainly a problem everywhere, but sometimes you get away with it and sometimes not. > > You can try running the code in an R session where you’ve enabled the valgrind debugger. > > My approach would be to start an R session in the terminal withR -d valgrindin then run the code lines from that example. It will make everything runmuchslower, so don’t be suprised if loading a package takes 5 minutes. It will get there eventually, and should detect any problem. > > There’s a nice intro to valgrind athttps://kevinushey.github.io/blog/2015/04/05/debugging-with-valgrind/

Pierre-Luc Germain (11:48:09) (in thread): > thanks for the answer, I had never heard about valgrind! > Unfortunately I’ve been completely unable to reproduce the error anywhere else than in the Bioc build machines (everything runs smoothly for me and everyone who tried on a variety of servers) , which makes your suggestion difficult…

Mike Smith (12:04:09) (in thread): > I’m suggesting that you try that even on a machine where it apparently works fine. With out-of-memory errors, they can sometimes just work fine because you get lucky with the bit of invalid memory it tries to access, but fundamentally there’s still a problem. Valgrind should still detect this and let you know; that’ll at least confirm the problem and maybe give you a hint as to where the problem lies (or the maintainer of Rsubread).

Mike Smith (12:08:35) (in thread): > If it’s any help, I get the error on my local machine (both with and without valgrind). The valgrind output is: > > || Process BAM file example1.bam... || > ==146058== Thread 4: > ==146058== Invalid read of size 8 > ==146058== at 0x4926D150: ArrayListPush (hashtable.c:117) > ==146058== by 0x4929C2BD: vote_and_add_count (readSummary.c:5208) > ==146058== by 0x4929D744: process_line_buffer (readSummary.c:3729) > ==146058== by 0x4929D744: process_line_buffer (readSummary.c:3173) > ==146058== by 0x4929E111: process_pairer_output (readSummary.c:2930) > ==146058== by 0x4927DB1D: SAM_pairer_do_one_BIN (input-files.c:3876) > ==146058== by 0x492806EF: SAM_pairer_do_next_read (input-files.c:3915) > ==146058== by 0x492808AD: SAM_pairer_thread_run (input-files.c:4580) > ==146058== by 0x5751608: start_thread (pthread_create.c:477) > ==146058== by 0x4E7E292: clone (clone.S:95) > ==146058== Address 0x10 is not stack'd, malloc'd or (recently) free'd > ==146058== > > ***** caught segfault ***** > address 0x10, cause 'memory not mapped' >

Pierre-Luc Germain (12:10:14) (in thread): > yes, finally!

Pierre-Luc Germain (12:13:49) (in thread): > thank you, with this I can probably find the offending bit in the data… (still don’t know why it’s not giving an error in 8 different mac/linux setups now!)

Hervé Pagès (13:31:41) (in thread): > microbiomeDataSets is in Suggests. Does the SPB install packages that are in Suggests?

Lori Shepherd (13:35:11) (in thread): > @Hervé PagèsIt should… but I’m actually surprised that it wasn’t found on tokay2 since its available via build report for experiment data packages – I temporarily installed it into the package specific package installation directory the SPB creates for each issue number to move forward

Hervé Pagès (13:56:21) (in thread): > microbiomeDataSets is not on tokay2 (https://bioconductor.org/checkResults/3.13/bioc-LATEST/tokay2-R-instpkgs.html), and since tokay2 does not do experiment data builds, it won’t get installed by the BBS. So the SPB would need to install it.

Lori Shepherd (14:04:17) (in thread): > The SPB does do suggests so not exactly sure what happened – I’ll check the logs to see if it failed for some reason or other

Hervé Pagès (14:13:14) (in thread): > FWIW I can also reproduce this on my Ubuntu 20.10 laptop: > > library(diffUTR) > example(countFeatures) > # ... > # ... > # cntFtr> bam_files <- list.files(system.file("extdata", package="diffUTR"), > # cntFtr+ pattern="bam$", full=TRUE) > # > # cntFtr> se <- countFeatures(bam_files, bins, isPairedEnd=TRUE) > # > # ***** caught segfault ***** > # address 0x10, cause 'memory not mapped' > # > # Traceback: > # 1: Rsubread::featureCounts(bamfiles, ..., tmpDir = tmpDir, annot.ext = binsframe, isGTFAnnotationFile = FALSE, strandSpecific = strandSpecific, allowMultiOverlap = allowMultiOverlap, useMetaFeatures = FALSE) > # 2: countFeatures(bam_files, bins, isPairedEnd = TRUE) > # 3: eval(ei, envir) > # 4: eval(ei, envir) > # 5: withVisible(eval(ei, envir)) > # 6: source(tf, local, echo = echo, prompt.echo = paste0(prompt.prefix, getOption("prompt")), continue.echo = paste0(prompt.prefix, getOption("continue")), verbose = verbose, max.deparse.length = Inf, encoding = "UTF-8", skip.echo = skips, keep.source = TRUE) > # 7: example(countFeatures) > # > # Possible actions: > # 1: abort (with core dump, if enabled) > # 2: normal R exit > # 3: exit R without saving workspace > # 4: exit R saving workspace > > > > still don’t know why it’s not giving an error in 8 different mac/linux setups now! > Are you sure you guys are using the right version of R/Bioconductor? Remember that you’re submitting your package for inclusion in BioC 3.13 so you need R 4.1. Here’s mysesssionInfo()(before the crash):

Hervé Pagès (14:13:14) (in thread): > > > sessionInfo() > R Under development (unstable) (2021-03-08 r80083) > Platform: x86_64-pc-linux-gnu (64-bit) > Running under: Ubuntu 20.10 > > Matrix products: default > BLAS: /home/hpages/R/R-4.1.r80083/lib/libRblas.so > LAPACK: /home/hpages/R/R-4.1.r80083/lib/libRlapack.so > > locale: > [1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C > [3] LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8 > [5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8 > [7] LC_PAPER=en_US.UTF-8 LC_NAME=C > [9] LC_ADDRESS=C LC_TELEPHONE=C > [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C > > attached base packages: > [1] stats graphics grDevices utils datasets methods base > > other attached packages: > [1] diffUTR_0.99.22 > > loaded via a namespace (and not attached): > [1] Rtsne_0.15 colorspace_2.0-0 > [3] rjson_0.2.20 hwriter_1.3.2 > [5] ellipsis_0.3.1 circlize_0.4.12 > [7] XVector_0.31.1 GenomicRanges_1.43.3 > [9] GlobalOptions_0.1.2 clue_0.3-58 > [11] rstudioapi_0.13 ggrepel_0.9.1 > [13] bit64_4.0.5 AnnotationDbi_1.53.1 > [15] fansi_0.4.2 xml2_1.3.2 > [17] codetools_0.2-18 splines_4.1.0 > [19] doParallel_1.0.16 cachem_1.0.4 > [21] geneplotter_1.69.0 SEtools_1.5.1 > [23] jsonlite_1.7.2 Rsamtools_2.7.1 > [25] Cairo_1.5-12.2 annotate_1.69.2 > [27] cluster_2.1.1 dbplyr_2.1.0 > [29] png_0.1-7 compiler_4.1.0 > [31] httr_1.4.2 lazyeval_0.2.2 > [33] assertthat_0.2.1 Matrix_1.3-2 > [35] fastmap_1.1.0 limma_3.47.10 > [37] prettyunits_1.1.1 tools_4.1.0 > [39] gtable_0.3.0 glue_1.4.2 > [41] GenomeInfoDbData_1.2.4 dplyr_1.0.5 > [43] rappdirs_0.3.3 V8_3.4.0 > [45] Rcpp_1.0.6 Biobase_2.51.0 > [47] vctrs_0.3.6 Biostrings_2.59.2 > [49] nlme_3.1-152 rtracklayer_1.51.5 > [51] iterators_1.0.13 stringr_1.4.0 > [53] openxlsx_4.2.3 lifecycle_1.0.0 > [55] ensembldb_2.15.2 restfulr_0.0.13 > [57] statmod_1.4.35 XML_3.99-0.6 > [59] DEXSeq_1.37.0 edgeR_3.33.3 > [61] zlibbioc_1.37.0 scales_1.1.1 > [63] TSP_1.1-10 ProtGenerics_1.23.7 > [65] hms_1.0.0 MatrixGenerics_1.3.1 > [67] parallel_4.1.0 SummarizedExperiment_1.21.1 > [69] AnnotationFilter_1.15.0 RColorBrewer_1.1-2 > [71] ComplexHeatmap_2.7.7 yaml_2.2.1 > [73] curl_4.3 memoise_2.0.0 > [75] ggplot2_3.3.3 biomaRt_2.47.5 > [77] stringi_1.5.3 RSQLite_2.2.4 > [79] randomcoloR_1.1.0.1 genefilter_1.73.1 > [81] S4Vectors_0.29.9 BiocIO_1.1.2 > [83] foreach_1.5.1 seriation_1.2-9 > [85] GenomicFeatures_1.43.6 BiocGenerics_0.37.1 > [87] filelock_1.0.2 zip_2.1.1 > [89] BiocParallel_1.25.5 Rsubread_2.5.6 > [91] shape_1.4.5 GenomeInfoDb_1.27.8 > [93] rlang_0.4.10 pkgconfig_2.0.3 > [95] matrixStats_0.58.0 bitops_1.0-6 > [97] lattice_0.20-41 purrr_0.3.4 > [99] GenomicAlignments_1.27.2 bit_4.0.4 > [101] tidyselect_1.1.0 magrittr_2.0.1 > [103] DESeq2_1.31.16 R6_2.5.0 > [105] IRanges_2.25.6 generics_0.1.0 > [107] DelayedArray_0.17.9 DBI_1.1.1 > [109] mgcv_1.8-34 pillar_1.5.1 > [111] survival_3.2-10 KEGGREST_1.31.1 > [113] RCurl_1.98-1.3 tibble_3.1.0 > [115] crayon_1.4.1 utf8_1.2.1 > [117] BiocFileCache_1.15.1 GetoptLong_1.0.5 > [119] progress_1.2.2 locfit_1.5-9.4 > [121] grid_4.1.0 sva_3.39.0 > [123] data.table_1.14.0 blob_1.2.1 > [125] digest_0.6.27 xtable_1.8-4 > [127] openssl_1.4.3 stats4_4.1.0 > [129] munsell_0.5.0 registry_0.5-1 > [131] askpass_1.1 >

Hervé Pagès (14:13:14) (in thread): > Gosh, that’s a lot of dependencies! Are you sure you need all that?

Pierre-Luc Germain (14:21:27) (in thread): > yes I’m running it with R 4.1 and Bioc 3.13 (though I admit that most others that tested were still on 3.12).

Pierre-Luc Germain (14:22:58) (in thread): > Icertainly don’t need all that, it just goes really fast with this pyramidal stuff…

Pierre-Luc Germain (14:27:44) (in thread): > ensembldb= 60 in one go…

Pierre-Luc Germain (14:27:50) (in thread): > anyway thanks a lot for the reports

Hervé Pagès (15:19:13) (in thread): > Right but also your own SEtools brings in 100+ deps and is used only forSEtools::sechm(). A more granular approach would help keep things a little bit lighter e.g. maybe split SEtools into the merging/melting/aggregation functions on one side and the plotting functions on the other side.

Pierre-Luc Germain (15:24:32) (in thread): > good idea, and thanks for the feedback

2021-03-23

Lambda Moses (23:04:06): > @Lambda Moses has joined the channel

2021-03-25

Mike Smith (05:25:17): > @Hervé PagèsI notice that GenomicFeatures is failing to build in devel, and must have been doing so for a long time as there isn’t a source tarball available. This breaks the installation of quite a few packages outside of the build system. Looking at thebuild logit seems something is messed up with the biomaRt cache. It’s finding an entry for the query in the GenomicFeatures example, but then complaining that whatever it finds isn’t an RDS file. I don’t know how it can get into this state, but I’ll add a try/catch to the cache reading function in biomaRt. As a quick fix it might be worth doing abiomaRt::biomartCacheClear()on the linux builder if there’s a persistent cache that’s got corrupted.

Mike Smith (09:59:40) (in thread): > I’ve pushed a change to biomaRt that tries to address this. It should delete the offending cache entry and then run the query as normal. It might be worth waiting one build cycle to see if it actually works!

Hervé Pagès (11:57:13) (in thread): > I’m trying to understand what could have happened to this file. The file is from March 1: > > biocbuild@nebbiolo1:~/.cache/biomaRt$ ls -al e7a9c3146b157_filee7a9cad72f86 > -rw-rw-r-- 1 biocbuild biocbuild 632763 Mar 1 21:30 e7a9c3146b157_filee7a9cad72f86 > > This predates the 3.13 builds on nebbiolo1 so it was likely generated by the 3.12 builds (nebbiolo1 only started to run the 3.13 builds last week, before that it was running the 3.12 builds). > Then: > > > bfc[["BFC126"]] > BFC126 > "~/.cache/biomaRt/e7a9c3146b157_filee7a9cad72f86" > > readRDS(bfc[["BFC126"]]) > Error in readRDS(bfc[["BFC126"]]) : unknown input format > > name <- load(bfc[["BFC126"]]) > > head(get(name)) > ensembl_transcript_id transcript_biotype chromosome_name strand > 1 Y110A7A.10.1 protein_coding I 1 > 2 Y110A7A.10.2 protein_coding I 1 > 3 F27C8.1.1 protein_coding IV -1 > 4 F27C8.1.2 protein_coding IV -1 > 5 F07C3.7.1 protein_coding V -1 > 6 F52H2.2a.1 protein_coding X 1 > transcript_start transcript_end > 1 5107833 5110183 > 2 5107843 5110164 > 3 9599178 9601695 > 4 9598986 9601695 > 5 9244402 9246360 > 6 2554231 2557726 > > Could it be that the caching mechanism in biomaRt switched from using.rdato using.rdsbetween release and devel?

Mike Smith (12:07:25) (in thread): > Let me look at the git history. I thought I’d always usedsaveRDSbut maybe I usedsaveat some point.

Mike Smith (12:15:43) (in thread): > Yep, I refactored a lot of this, but apparently it used to usesave()Here’s the line inRELEASE_3_12https://github.com/grimbough/biomaRt/blob/607b632829ac285abf1d81e7abbc8ebc41c045b0/R/biomaRt.R#L642 - Attachment: R/biomaRt.R:642 > > save(result, file = tf) >

Mike Smith (12:20:02) (in thread): > Thanks for digging into the details of the file.

Hervé Pagès (12:51:16) (in thread): > Isn’t that going to be a problem for other users, not just the build system? It seems that the assumption here is that the biomaRt cache can be shared across BioC versions.

Hervé Pagès (13:06:46) (in thread): > Maybe just use > > read_rds_or_rda <- function(file) > { > res <- try(readRDS(file), silent=TRUE) > if (!inherits(res, "try-error")) > return(res) > env <- new.env(parent=emptyenv()) > name <- try(load(file, envir=env), silent=TRUE) > if (!inherits(name, "try-error")) > return(get(name, envir=env)) > stop("'", file, "' not an .rds or .rda file") > } > > instead ofreadRDS()?

Hervé Pagès (13:18:44) (in thread): > Actually biomaRt 2.47.6 should address these concerns and replacing these old cached files with fresh ones seems indeed cleaner. I didn’t remove the old file on nebbiolo1 so we’ll be able to see how that goes.

Hervé Pagès (13:26:21) (in thread): > FWIW~/.cache/biomaRt/contains 263 files and 223 of them are old (i.e. from the 3.12 builds). All of them should get an automatic refresh tonight.

Hervé Pagès (14:47:20) (in thread): > One more thing. This problem currently affects the 18 following software packages on nebbiolo1: > {badregionfinder} > cTRAP > easyRNASeq > FELLA > GenomicFeatures > GenVisR > metaseqR2 > MGFR > OrganismDbi > Pigengene > PPInfer > PSICQUIC > R453Plus1Toolbox > RnaSeqSampleSize > SeqGSEA > Sushi > SWATH2stats > yarn >

Mike Smith (16:06:30) (in thread): > Fingers cross the fix works. The cache includes the ensembl version in the hashing, but not the package version. If this fails then I’ll add that to the hash function, which should make it resilient to this. Seems like a an example of why having the devel branch benefits users, as we’ve discovered this problem without anyone actually reporting it.

2021-03-26

Hervé Pagès (13:32:11) (in thread): > All these packages are green on today ’s report:https://bioconductor.org/checkResults/3.13/bioc-LATEST/Well done:sunglasses:

Mike Smith (16:54:05) (in thread): > Awesome. Thanks for the help diagnosing the issue.

2021-03-31

Lisa Cao (12:49:47): > @Lisa Cao has joined the channel

2021-04-12

Lori Shepherd (14:43:55): > Tentative 3.13release scheduleposted. Please check website for important deadlines

2021-04-13

Nicholas Cooley (14:00:32): > Hi I’ve got a weird question; I changed my git user name locally recently and commits post that change do not appear to have registered with bioconductor (though they work just fine with my repository), do i need to change something with my key? > > Shory story: I switched work computers at one point and never changed the login on the new one, but when i transferred my ssh key everything worked with bioconductor, now that i’ve changed the login, commits don’t seem to be registering. So I’ve changed the local git login to be the one that the repository was originally created with, and now it’s not working?

Nitesh Turaga (14:01:52): > I maintain the git ecosystem, and I have to say, I do not understand what is happening. Can you rephrase ?

Nitesh Turaga (14:02:17): > The easiest solution is always, just create a new key and add it to your BiocCredentials account if you move machines.

Nitesh Turaga (14:02:27): > Solves most problems, rather than copying keys around.

Nicholas Cooley (14:07:15): > yeah, i wanted to copy around keys because that solution was easier for the OSG

Nicholas Cooley (14:09:24): > So: created a bioconductor package on the old machine that had the github login credentials. Change machines, and moved ssh key, didn’t change update github credentials, everything worked fine. Updated github credentials, now commits aren’t registering on the upstream remote.

Hervé Pagès (14:11:46): > Not really a build question. The#bioc_gitchannel is better for this kind of question.

Nicholas Cooley (14:14:43) (in thread): > ohp, thanks!

2021-04-14

FelixErnst (10:19:26): > The software builds were not update for devel the last 2-3 days (the report was updated last on the 11th of April). Any news, when the next build will go through?

Lori Shepherd (10:29:13): > thank you for making us aware . we will look into it

FelixErnst (10:29:26) (in thread): > Thx!

Lori Shepherd (10:53:25): > There were some ERROR’s generating the builds for Tuesday. I just checked the current logs and everything appears to be running smoothly for today. I expect today’s report to post later today but if it does not we will continue to investigate

2021-04-16

Nicholas Cooley (14:37:49) (in thread): > Are you referring to the error asking to include rmarkdown in the description file? That was a bit unexpected.

Lori Shepherd (14:42:15) (in thread): > no. that was not what I was referring too. It is unexpected but unfortunately ERROR’s package maintainers will have to fix

2021-04-20

Hervé Pagès (12:43:59): > <!channel>knitr1.32.4 contains a fix for the currentR CMD builderror that affects hundreds of packages on our daily builds (the “r/markdown package should be declared as a dependency” error). It’s not on CRAN yet but we’ve installed it from GitHub on all the build machines. Hopefully the error will go away on tomorrow’s build report.

Dan Bunis (12:45:21): > Suppose that’s why there hadn’t been a devel build report since Sat?

Nicholas Cooley (12:46:30) (in thread): > Will there be any harm in leaving both markdown and rmarkdown in theSuggestsline?

Dan Bunis (12:46:42): > jk… I see one for today

Hervé Pagès (12:48:39) (in thread): > Yihui’s recommendation still applies so, yes, you need this in your package if you’re usingmarkdownorrmarkdown.

Hervé Pagès (12:50:56) (in thread): > I was away and didn’t have an opportunity to fully investigate this but it seems that this was a different issue. Something went wrong with Mac builder machv2.

Nicholas Cooley (13:31:56) (in thread): > Oh, i guess I am a little confused, when my package errored on Bioconductor it wasn’t erroring in my local builds and checks, but I’m still building in the current R and not R devel. They’ll stay in and will be updated with the next push.

2021-04-21

Aaron Lun (03:56:53): > well, this is kind of surreal. > > > BiocManager::install("snifter") > Bioconductor version 3.13 (BiocManager 1.30.12), ?BiocManager::install for help > Bioconductor version 3.13 (BiocManager 1.30.12), R Under development (unstable) > (2021-01-24 r79876) > Installing package(s) 'snifter' > Old packages: 'reticulate', 'S4Vectors' > Update all/some/none? [a/s/n]: n > Warning message: > package ‘snifter’ is not available for this version of R >

FelixErnst (04:04:37) (in thread): > Hi. The issue about the windows specific warnings for generated html files with apparently invalid file names (containing the<-) is still a mystery to me (http://bioconductor.org/checkResults/devel/bioc-LATEST/Modstrings/riesling1-checksrc.html). This seems to a BBS specific issue (not seen on CRAN and GHA). Is there anything I can do for helping to solve the problem?

Lori Shepherd (06:58:15) (in thread): > Yes it has to do with commands that were shared with us to not rebuild. Please try the specific commands for each step to reproduce as seen in the summary of the build report? for installmkdir Modstrings.buildbin-libdir && D:\biocbuild\bbs-3.13-bioc\R\bin\R.exe CMD INSTALL --merge-multiarch --build --library=Modstrings.buildbin-libdir Modstrings_1.7.1.tar.gz && D:\biocbuild\bbs-3.13-bioc\R\bin\R.exe CMD INSTALL Modstrings_1.7.1.zip && rm Modstrings_1.7.1.tar.gz Modstrings_1.7.1.zipfor buildchmod a+r Modstrings -R && D:\biocbuild\bbs-3.13-bioc\R\bin\R.exe CMD build --keep-empty-dirs --no-resave-data Modstringsand for check`D:-bioc.exe CMD check –force-multiarch –install=check:Modstrings.install-out.txt –library=D:-bioc–no-vignettes –timings Modstrings_1.7.1.tar.gz``

FelixErnst (07:20:18) (in thread): > I will try to reproduce the error. Can you give me a hint, where themeat/directory comes from? The html files are expected to be inD:/biocbuild/bbs-3.13-bioc/meat/*.buildbin-libdir/, but I don’t see this subdirectory in your commands and also didn’t have any luck finding it inRenviron.bioc

Lori Shepherd (07:30:21) (in thread): > yeah slightly confusing … the meat directory stores all the build product results I believe of running install, build, and check – > > biocbuild@malbec2:~/bbs-3.13-bioc/meat$ ls Modstrings* > Modstrings_1.7.1.tar.gz Modstrings.checksrc-summary.dcf > Modstrings.buildsrc-out.txt Modstrings.install-out.txt > Modstrings.buildsrc-summary.dcf Modstrings.install-summary.dcf > Modstrings.checksrc-out.txt > > Modstrings: > codecov.yml DESCRIPTION man NEWS R tests > data inst NAMESPACE _pkgdown.yml README.md vignettes > > Modstrings.Rcheck: > 00check.log 00install.out Modstrings-Ex.Rout Modstrings-Ex.timings tests >

FelixErnst (07:30:59) (in thread): > I’ll give it a try

FelixErnst (08:04:36) (in thread): - File (Plain Text): Unbenannt

FelixErnst (08:04:46) (in thread): > So I gave it a try with the alpha release using the offending commandD:\biocbuild\bbs-3.13-bioc\R\bin\R.exe CMD check --force-multiarch --install=Modstrings.install-out.txt --library=D:\biocbuild\bbs-3.13-bioc\R\library --no-vignettes --timings Modstrings_1.7.1.tar.gz. I didn’t get~any~the error.

FelixErnst (08:06:46) (in thread): > (Just latex error/warning at the end)

FelixErnst (08:08:04) (in thread): > So I cannot reproduce it. From my point of view it is either a permission issue on the BBS or fixed inr80202, but not inr80146

Alan O’C (08:09:39) (in thread): > Huh, build reports are okay

Marcel Ramos Pérez (10:55:17) (in thread): > The source package is missing:male-detective:http://bioconductor.org/packages/devel/bioc/html/snifter.html

Alan O’C (10:55:43) (in thread): > :shrug:

Marcel Ramos Pérez (10:59:40) (in thread): > We’ll take a look

Marcel Ramos Pérez (12:48:38) (in thread): > Hervé took a closer look and it appears we had some errors with indexing the PACKAGES file but the source is available via direct linkhttps://bioconductor.org/packages/3.13/bioc/src/contrib/snifter_1.1.4.tar.gz. Hervé is on it. Thanks for the catch!

Hervé Pagès (12:59:07) (in thread): > FWIW the--install=check:/path/to/install/output/logtrick was suggested to us by the CRAN folks (unfortunately it’s undocumented) and we’ve been using it for many years without any problem. These warnings only appeared a few months ago in the BioC 3.13 builds so they seem related to some change in R-devel that triggers them but only whenR CMD checkis used with the--install=check:/path/to/install/output/logtrick.

FelixErnst (13:01:52) (in thread): > Ah ok. I changed the install argument’s value to match what I have as a setup. I will check again tomorrow with thecheck:part

Hervé Pagès (13:14:52) (in thread): > Install the package normally, no need to use the complicated command that we use on the Windows builder for that. But you need to capture the output ofR CMD INSTALLby redirecting it to a file with>(yes this works on Windows!), sayModstrings.install-out.txt. Then feed that file toR CMD checkwith--install=check:Modstrings.install-out.txt. This wayR CMD checkwon’t try to re-install the package because the file you gave it is all it needs to determine whether the package can successfully be installed or not.

Hervé Pagès (13:22:45) (in thread): > I have to postpone that a little sorry because there’s another fire that requires urgent intervention. It’s related to today’s builds and some disk space issues on machv2.

Hervé Pagès (17:24:30) (in thread): > Thissnifter_1.1.4.tar.gzsource tarball is actually empty which is whytools::write_PACKAGES()skipped it.snifteris not an isolated case: looks like we have a few other empty tarballs in the 3.13 repo. Not sure how we could end up in this situation.:thinking_face:Still investigating…

Alan O’C (17:29:04) (in thread): > The plot, as they say, thickens

Hervé Pagès (17:36:26) (in thread): > I know we also had some disk space issues about one month ago on nebbiolo1 but these empty tarballs were produced a few days after we addressed them. And if it was the case that they were somehow produced during the builds but truncated because of lack of disk space,R CMD checkwould have failed so they wouldn’t have been allowed to propagate. > The suspense is becoming unbearable…

Hervé Pagès (17:41:47) (in thread): > Just making sure thatR CMD checkis not playing tricks on us. Using the empty tarball: > > biocbuild@nebbiolo1:~/sandbox$ ~/bbs-3.13-bioc/R/bin/R CMD check snifter_1.1.4.tar.gz > * using log directory '/home/biocbuild/sandbox/snifter.Rcheck' > * using R Under development (unstable) (2021-04-06 r80146) > * using platform: x86_64-pc-linux-gnu (64-bit) > * using session charset: UTF-8 > * checking package directory ... ERROR > package directory '/home/biocbuild/sandbox/snifter.Rcheck/00_pkg_src/snifter' does not exist > * DONE > > Status: 1 ERROR > See > '/home/biocbuild/sandbox/snifter.Rcheck/00check.log' > for details. > > Good! (even though the error message could be a little bit more explicit)

Hervé Pagès (18:04:41) (in thread): > I give up. > > Here is the final record on file: On the night of March 27-28,R CMD buildsomehow managed to produce 5 empty source tarballs in the context of the daily builds (densvis_1.1.10.tar.gz,DepecheR_1.7.2.tar.gz,ORFik_1.11.23.tar.gz,PhyloProfile_1.5.10.tar.gz,snifter_1.1.4.tar.gz). Later in the morning of March 28, these tarballs managed to propagate tobioconductor.orgusing some unknown artifice to trick the guards posted at various checkpoints. > Case closed. > > I removed these tarballs so whatever tarballs end up being produced by today’s builds will replace them tomorrow (granted they’re allowed to propagate). I’ll also add an additional check that empty tarballs are not allowed to propagate. (I’d hope I didn’t have to perform this kind of silly check but such is life.)

Alan O’C (18:09:01) (in thread): > Sometimes life’s great mysteries are not meant to be solved:shrug:

2021-04-22

Hervé Pagès (15:51:03) (in thread): > I think we’re good today. These 5 tarballs got propagated and are now available viaBiocManager::install(). > > PS: As always Urban Dictionary is the best:https://www.urbandictionary.com/define.php?term=Snifter:wink:

2021-04-23

FelixErnst (03:03:22) (in thread): > And that is the problem. The R CMD check detects that there are errors/warnings, since theModstrings.install-out.txtcontains this after install:

FelixErnst (03:03:51) (in thread): - File (Plain Text): Modstrings.install-out.txt

FelixErnst (03:12:04) (in thread): > So the GHA setup I use Is probably not reproducing this, since it relies on thercmdcheckwrapper, which does not install separately, but builds and checks (https://github.com/FelixErnst/Modstrings/runs/2038313713).

FelixErnst (03:15:54) (in thread): > And if I just compare the BBS output, the difference is quite simply, that the output of the install step is much more verbose fordevel(http://bioconductor.org/checkResults/3.12/bioc-LATEST/Modstrings/tokay1-checksrc.htmlvshttp://bioconductor.org/checkResults/devel/bioc-LATEST/Modstrings/riesling1-checksrc.html).

Hervé Pagès (12:36:25) (in thread): > Yeah, right, all these Rd warnings are emitted during INSTALL:https://bioconductor.org/checkResults/3.13/bioc-LATEST/Modstrings/riesling1-install.htmlI don’t know how we could miss this before. So do you also get these Rd warnings when you installModstringson your Windows box? In which case we could conclude that the problem is real and it would need to be reported to the R folks.

FelixErnst (12:38:01) (in thread): > Yes they are also emited duringinstall.packages(..., type = "source", repos = NULL)on the current R 4.1.0 alpha

Hervé Pagès (12:39:19) (in thread): > GOOD! (and bad at the same time) But it’s a relief for me. Thanks Felix! Do you want to bring to the R-devel list?

FelixErnst (12:40:00) (in thread): > I have been looking at build reports on CRAN.https://cran.r-project.org/web/checks/check_results_Matrix.htmlshows the same kind of warnings

Hervé Pagès (12:40:34) (in thread): > EXCELLENT! That’ll make the task easier:wink:

FelixErnst (12:40:50) (in thread): > but this time it is on linux with different offending characters

Hervé Pagès (12:42:26) (in thread): > Ouch! But what about Windows? Maybe we should just focus on theR CMD INSTALLRd warnings on Windows for now.

FelixErnst (12:43:04) (in thread): > Do you/anyone know a package defining a S4 replace function which is on CRAN?

Hervé Pagès (12:46:15) (in thread): > I don’t. Another difficulty is that the CRAN check reports don’t directly show the output ofR CMD INSTALL. We only get to see it ifR CMD checkis willing to show it to us, typically whenR CMD checktries to install the package and fails.

FelixErnst (12:47:09) (in thread): > I noticed that as well.Matrixwas a lucky find, since it the only “S4” package I know of on CRAN

Hervé Pagès (12:50:15) (in thread): > Anyways that shouldn’t stop us. If you want to be totally confident that you actually found something worth reporting, we should get someone else runR CMD INSTALLonModstrings(or any other BioC package that triggers these warnings) on their Windows box. But maybe it’s not necessary?

FelixErnst (12:50:36) (in thread): > I am writing a draft. 5 min

FelixErnst (13:01:16) (in thread): > Send!

Hervé Pagès (13:01:27) (in thread): > I would provide the links to the INSTALL commands on Windows. The fact that those warnings propagate toR CMD checkon our build machines is due to our use of the--install=check:/path/to/install/output/logtrick and, while interesting, it’s a little bit distracting.

FelixErnst (13:01:46) (in thread): > Yeah lets see how it goes

Hervé Pagès (13:01:52) (in thread): > Ah you sent it. I stop commenting your draft then…

FelixErnst (13:02:24) (in thread): > I have to go, thats why I was a bit impatient. Sry

FelixErnst (13:02:26) (in thread): > :slightly_smiling_face:

Hervé Pagès (13:02:44) (in thread): > No worries. Thx for helping with this!

FelixErnst (13:04:20) (in thread): > The install.out is posted below on the BBS output, so I think the output is complete, but in the wrong order (check before install).

Hervé Pagès (13:06:13) (in thread): > I don’t see your post on R-devel yet but for some reason it usually takes a while between the moment a post is sent to the list and the moment it shows up in my inbox.

FelixErnst (13:11:15) (in thread): > it is the moderation queue

Hervé Pagès (13:13:25) (in thread): > I don’t know, I thought this list moderates only posts from non-members. You’re a member of the list right?

FelixErnst (13:13:52) (in thread): > nope:slightly_smiling_face:my inbox is full enough

Hervé Pagès (13:14:42) (in thread): > well mine is too but I use filters

Hervé Pagès (13:15:26) (in thread): > so things like R-devel stuff goes directly in their little dedicated folder

Hervé Pagès (13:16:35) (in thread): > but I can’t blame you for not being eager to join the R-devel list

FelixErnst (13:16:44) (in thread): > I know, I know, should have done it a long time ago.

2021-04-24

FelixErnst (16:07:23) (in thread): > Posted and confirmed on R-devel. Bug posted in Bugtracker.

2021-04-25

Hervé Pagès (03:37:42): > <!channel>We’ve made 2 changes to the build weekly schedule: > * Data experiment builds now run 3 times a week (instead of 2): on Tuesdays, Thursdays, and Saturdays > * No more software builds on Saturdays (this means no report on Sundays). This gives us plenty of room to run the data-experiment and long tests builds that day. > These changes will be effective starting next Monday and are for the 3.13 builds for now (the 3.14 builds will likely follow that new schedule too).

FelixErnst (12:27:51) (in thread): > Should be fixed inr80222

Hervé Pagès (13:35:43) (in thread): > Thanks Felix. We’re going to update R to R alpha on the build machines next week so all these spuriousR CMD checkwarnings should go away.

2021-04-29

Hervé Pagès (17:48:01): > 362 packages break on machv2 (Mac builder) today with apolygon edge not founderror

Aaron Lun (17:48:50): > macs and visual design, a notoriously bad combination

Hervé Pagès (17:50:15): > you’re being nice, I would say “macs and scientific computing”…

2021-05-03

FelixErnst (04:28:32) (in thread): > Hi Herve, did you have a chance to look into updated to R alpha also for riesling1?

2021-05-04

Steffen Neumann (15:34:28): > Hi, would anyone know a docker container with R-4.1 and clang-1[12] ? I need to reproduce a build error on a platform I don’t have. I was nearly there withdocker run --rm -it rhub/debian-clang-devel /opt/R-devel/bin/R -e 'install.packages("BiocManager", repos="https://cloud.r-project.org"); BiocManager::install("Biobase")'but that has R-devel installed, andbiocmanager::install()complains.

Nitesh Turaga (15:37:04): > which package are you trying to reproduce errors for?

Steffen Neumann (15:39:07) (in thread): > mzR on clang 11 and 12

Steffen Neumann (15:40:53) (in thread): > https://github.com/sneumann/mzR/issues/244

Hervé Pagès (21:20:24) (in thread): > Don’t know if this would work on our Docker image but FWIW here’s what I did on my laptop (Ubuntu 20.10): > > sudo apt install clang-11 > > export CC=/usr/bin/clang-11 > export CPP=/usr/bin/clang-cpp-11 > export CXX=/usr/bin/clang++-11 > > Download, configure and compile latest R-4.1 as usual. Then from R: > > install.packages("BiocManager") > BiocManager::install("mzR") > > and after a few minutes, I got this error: > > ... > pwiz/data/common/BinaryIndexStream.cpp:159:15: error: > cannot assign to non-static data member within const member function > 'operator[]' > next_ = *Off * value_size*; > ~~~~~~~~~ ^ > pwiz/data/common/BinaryIndexStream.cpp:157:12: note: > member function 'pwiz::data::(anonymous > namespace)::stream_vector_const_iterator::operator[]' is declared const > here > reference operator[](difference_type _Off) const > ~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > 1 warning and 1 error generated. > make: ***** [/home/hpages/R/R-4.1.r80263-clang11/etc/Makeconf:175: pwiz/data/common/BinaryIndexStream.o] Error 1 > ERROR: compilation failed for package 'mzR' > * removing '/home/hpages/R/R-4.1.r80263-clang11/library/mzR' > > The downloaded source packages are in > '/tmp/RtmphkZhJ3/downloaded_packages' > Updating HTML index of packages in '.Library' > Making 'packages.html' ... done >

2021-05-05

FelixErnst (03:00:55) (in thread): > @Hervé PagèsThx for updating the builder. The warnings are gone

2021-05-07

Hervé Pagès (14:02:15) (in thread): > Great, I forgot to check that. Thanks for checking!

2021-05-11

Megha Lal (16:43:47): > @Megha Lal has joined the channel

2021-05-12

Devon Kohler (18:10:43): > @Devon Kohler has joined the channel

2021-05-13

Hervé Pagès (00:10:17): > <!channel>: Help needed! All the BioC software packages that depend directly or indirectly onrglfail to install at the moment on our macOS builder. They all fail with an error that doesn’t say much about what’s going on. See for examplehttps://bioconductor.org/checkResults/3.13/bioc-LATEST/DaMiRseq/machv2-install.htmlThe complete list of packages affected by this is:clippda,DaMiRseq,fCI,GARS,iCheck,mAPKL,MoonlightR,NormqPCR,OMICsPCA,optimalFlow,Rcade,RDRToolbox,TrajectoryGeometry,tscR, andTTMap. Can someone that is on a Mac reproduce this? > > FWIW our macOS builder machv2 uses the latestrglMac binary for 4.1 (rgl_0.106.8.tgz). Downgradingrglto 0.105.22 seems to solve the issue so Istronglysuspect that this is an issue with the latest versions ofrgl. But I’d like to make sure that the problem can be observed on other macOS systems before reporting to R-SIG-Mac. Thanks!

Charlotte Soneson (02:10:53) (in thread): > FWIW, I don’t seem to be able to reproduce this on my mac (Catalina 10.15.7, R 4.1.0 RC from May 10). I grabbed the source forMoonlightR1.17.0. > > charlotte$ R CMD install MoonlightR > * installing to library '/Library/Frameworks/R.framework/Versions/4.1/Resources/library' > * installing **source** package 'MoonlightR' ... > **** using staged installation > **** R > **** data > ***** moving datasets to lazyload DB > **** inst > **** byte-compile and prepare package for lazy loading > **** help > ***** installing help indices > **** building package indices > **** installing vignettes > **** testing if installed package can be loaded from temporary location > **** testing if installed package can be loaded from final location > **** testing if installed package keeps a record of temporary installation path > * DONE (MoonlightR) > charlotte$ R > > R version 4.1.0 RC (2021-05-10 r80288) -- "Camp Pontanezen" > Copyright (C) 2021 The R Foundation for Statistical Computing > Platform: x86_64-apple-darwin17.0 (64-bit) > > R is free software and comes with ABSOLUTELY NO WARRANTY. > You are welcome to redistribute it under certain conditions. > Type 'license()' or 'licence()' for distribution details. > > Natural language support but running in an English locale > > R is a collaborative project with many contributors. > Type 'contributors()' for more information and > 'citation()' on how to cite R or R packages in publications. > > Type 'demo()' for some demos, 'help()' for on-line help, or > 'help.start()' for an HTML browser interface to help. > Type 'q()' to quit R. > > > packageVersion("rgl") > [1] '0.106.8' > > packageVersion("MoonlightR") > [1] '1.17.0' >

Charlotte Soneson (02:12:47) (in thread): > I originally had an R 4.1 version from about a month or so ago (forgot to check exactly which one before reinstalling the recent RC), and it worked fine there as well.

Hervé Pagès (02:28:36) (in thread): > Damn, that’s what I was worried about! Looks like it could be some bad interaction between the latest versions ofrgland XQuartz on our Mac builders. I just found a recent post by Duncan Murdoch on R-SIG-Mac about MacOS updates and/or XQuartz updates breakingrglfor some people. Will need to investigate. Thanks@Charlotte Soneson!

2021-05-14

Jonathan Griffiths (03:40:16): > Do you think we could extend the build failure email notification to the Hub packages? A couple of times in this release my EHub package was broken by a change to one of its dependencies, which I didn’t notice until some (unknown!) time after. An email would have been handy!

Aaron Lun (04:05:53): > several build failures due to corrupted AHub cache on the builders

Michael Love (05:12:51): > @Michael Love has joined the channel

Michael Love (05:13:41) (in thread): > For me its only for Linux builder

Davide Risso (07:12:23): > @Davide Risso has joined the channel

Lori Shepherd (07:53:50) (in thread): > already taken care of should be fine on today’s report

Michael Love (07:54:19) (in thread): > :pray:

Lori Shepherd (07:54:55) (in thread): > I thought the experiment data packages also received build failure notifications but I’ll check and if not suggest it as an enhancement. This does seem like a good idea.

Ludwig Geistlinger (10:49:40): > @Ludwig Geistlinger has joined the channel

Hervé Pagès (10:52:03) (in thread): > So we’re only sending the automatic notifications for software packages and when the package fails on the Linux builder at the moment. I’m going to activate them for data experiment and workflow packages too.

koki (21:46:23): > My packages scTensor and scTGIF have ERROR in Daily Package Builder for BioC 3.13.https://master.bioconductor.org/checkResults/3.13/bioc-LATEST/scTensor/nebbiolo1-install.htmlhttps://master.bioconductor.org/checkResults/3.13/bioc-LATEST/scTGIF/nebbiolo1-install.htmlI know the reason completely; it is because the dependent package, rTensor, was suddenly removed from CRAN because the previous maintainer did not deal with the build error.https://cran.r-project.org/web/packages/rTensor/index.htmlI’m taking over the maintenance now, and I think it will be released again on CRAN in a few days, but I’m not sure if it will be in time for the BioC 3.13 release on May 20. > > I also have to re-submit another CRAN package nnTensor, to be more precise, I have to modify nnTensor after releasing rTensor because the dependency graph is as follows. > rTensor -> nnTensor -> scTensor/scTGIF > > Will scTensor/scTGIF be removed from BioC 3.13 for the above reasons?

Lori Shepherd (22:49:00): > we will keep them in for now. Please submit to CRAN the appropriate dependencies asap to avoid deprecation in 3.14

koki (23:52:52): > Thanks, Lori. Yes, I will submit nnTensor soon.

2021-05-15

Pedro Baldoni (04:51:18): > @Pedro Baldoni has joined the channel

2021-05-17

Jianhai Zhang (00:32:50): > Does the daily build occur at 3 PM eastern time?

Jianhai Zhang (01:27:41): > Does anyone have the same error? Even though I already registered.ERROR: Maintainer must register at the support site; visit``https://support.bioconductor.org/accounts/signup/

Jianhai Zhang (02:32:16): > Resolved!

Hervé Pagès (14:25:56) (in thread): > More or less. Seehttps://bioconductor.org/developers/how-to/troubleshoot-build-report/

Jianhai Zhang (19:18:43) (in thread): > Hi Herve, can you let me know today’s build results of my package spatialHeatmap if it is done?@Hervé Pagès

Jianhai Zhang (19:20:25) (in thread): > There is no error on my computer, no sure on the Bioc platforms.

Hervé Pagès (19:41:44) (in thread): > It’s not done yet. But in general I’d rather not do that unless there’s a very good reason. Can’t we just wait for the next report?

Jianhai Zhang (20:35:07) (in thread): > I pushed many updates to github yesterday. Even though no check errors on my computer, l don’t know what could happen on bioc machine.

Jianhai Zhang (20:35:57) (in thread): > Ok, I can wait for the report.

Jianhai Zhang (20:37:01) (in thread): > @Hervé Pagès

Hervé Pagès (22:43:47) (in thread): > export RGL_USE_NULL=TRUEis the workaround. Now I can finallyR CMD INSTALL MoonlightRon machv2:persevere:. Let’s see what happens on the build report after tomorrow.

2021-05-19

Hervé Pagès (14:16:42) (in thread): > The 15 packages affected by the nasty rgl issue are green on today’s report. Problem solved! > > If at leastrglcould say something about setting RGL_USE_NULL to TRUE instead of failing to load with a totally useless and misleading > > Error: package or namespace load failed for 'rgl': > .onLoad failed in loadNamespace() for 'rgl', details: > call: grDevices::quartz() > error: unable to create quartz() device target, given type may not be supported > In addition: Warning message: > In grDevices::quartz() : No displays are available > > it would have saved me hours of troubleshooting and we would have avoided months of false positive ERRORs on the daily reports.:angry:

2021-05-20

Lori Shepherd (15:23:05): > Bioconductor 3.13 is Released!! Thank you to all developers and community members for contributing to the project. See fullrelease announcement

2021-05-21

Guido Barzaghi (04:14:20): > @Guido Barzaghi has joined the channel

Dan Bunis (20:16:14): > Seurat annoyances continue:face_with_rolling_eyes:… Would it be possible to roll back Seurat on the BBS to 4.0.1? I can recreate this current dittoSeq buildissuelocally with Seurat-v4.0.2 installed. It’s in anas.Seuratcall. But if I roll back to v4.0.1, the issue goes away. I’ve reported this to the Seurat-teamhttps://github.com/satijalab/seurat/issues/4520as well.

Dan Bunis (20:18:55): > I already scrambled to eliminate the main Seurat workflow from the dittoSeq vignette last month after there was a bug in a different one of their “main workhorse” functions… just another reason that I’ll be reducing Seurat’s impact on dittoSeq’s build reqs before the next release!

Dan Bunis (20:23:30): > I wanted to report this before I forgot, but I’m not in a rush to push any updates at the moment. Great job with the release to the build team & I hope you all have a wonderful weekend!

Hervé Pagès (22:10:23): > Hi Dan, the build system tries hard to use the latest version of everything so if I go on riesling1 now to roll backSeuratto 4.0.1 the first thing the build system will do is updateSeuratagain to the latest version available on CRAN (4.0.2). Call it stubborn. Note that as soon as there’s a version ofSeuratthat addresses the problem we can manually install it on the build machines, we don’t need to wait that it becomes available on CRAN, but it would need to have a version >= 4.0.2 otherwise the build system will replace it again with the broken version. If the fix is simple, you could even forkSeuraton GitHub, fix it, bump its version (say to 4.0.2.1) and give us the link so we can install it on the builders. But that’s only if fixing thedittoSeq’s error on Windows is really important for you and cannot wait. Is it really?

Dan Bunis (23:01:31): > Gotcha. No, none of that sounds worth it to me lol. I don’t need to keep the function at all.

2021-05-25

Hervé Pagès (14:19:50): > <!channel>We’re considering the option of running the release builds (3.13 software builds) every other day e.g. builds would start on Sundays, Tuesday, and Thursdays, with the report being produced the day after (so on Mon, Wed, and Fri). The reason for this change is to try to reduce the amount of build power used for these builds. After all, release is expected to be quite stable so the rate of commits should be much lower there than in devel. Just to clarify nothing would change for the devel builds i.e. the 3.14 software builds will still run every day of the week except on Saturdays. Any strong feeling against this? Thanks

Steffen Neumann (14:40:23) (in thread): > I added the thumbdown just so people can more easily “vote”. Subtract one, I also did thumbs up:wink:

Hervé Pagès (15:15:58) (in thread): > Fraud! Our election is being stolen!:grin:

Steffen Neumann (15:16:13) (in thread): > Let’s recount ! We need to find XXX votes

2021-05-26

Mike Smith (08:00:21) (in thread): > This might be too big a change to the infrastructure, but might it be possible to switch release building to something that is reactive to a code change? I don’t know the volume of changes to the release branch, but I could imaging building everything once a week for completeness, but rebuilding packages that have changed on a daily (or maybe even hourly) basis. That way a developer still gets fairly rapid feedback if they’ve messed something up or deployment of a bug fix in response to an upstream CRAN package for example, but the resource requirements shouldn’t be too large if updates are infrequent.

Steffen Neumann (08:04:09) (in thread): > That came up a few times every now and then,, including by me:wink:It wouldn’t even have to be run by BioC, if it was encapsulated enough to beidenticalto the BioC builders … Alternatively, one could trigger that similar to the win-builder Service in Dortmund.

Henrik Bengtsson (15:46:29): > @Henrik Bengtsson has joined the channel

Hervé Pagès (17:12:29) (in thread): > Incremental builds are an option but require some work to the BBS code. Slowing down the frequency of the software builds in release is just the easy way to go for now and maybe a good enough compromise?

Henrik Bengtsson (18:52:40): > Correct me if I’m wrong, but Bioconductor doesn’t runR CMD checkwith*R_CHECK_INSTALL_DEPENDS*=true, correct? If not, I’d like to suggest doing so because it catches when one forgets to declare package dependencies. > > From ‘R Internals’: > > *R_CHECK_INSTALL_DEPENDS*If set to a true value and a test installation is to be done, this is done with.libPaths()containing just a temporary library directory and.Library. The temporary library is populated by symbolic links to the installed copies of all the Depends/Imports/LinkingTo packages which are not in.Library. Default:false(buttruefor CRAN submission checks). [edit: also enabled forR CMD check --as-cran] > I’m running reverse dependency checks onmatrixStats(~350 packages) and I’m (again) spotting several Bioconductor packages that forget to declare all packages needed in examples or tests underSuggests:. Just to clarify what I’m talking about, see for instancehttps://github.com/cbg-ethz/nempi/issues/1(one of several examples). I try to report upstream, but it’s quite tedious. > > Any package that don’t declare all the dependencies needed for examples and tests will fail (=ERROR) early and not benefit from reverse dependency checks, and vice versa. > > cc/@Dirk Eddelbuettel(since I know this is also something you’re caring about, particularly when revdep:ingRcpp)

Hervé Pagès (19:00:14): > We can try this and see how it goes. But I’d like to wait and see what happens with the new*R_CHECK_LENGTH_1_CONDITION*and*R_CHECK_LENGTH_1_LOGIC2*settings discussed earlier first. So maybe we can wait a week or two before adding the*R_CHECK_INSTALL_DEPENDS*=truesetting. I’m putting this on the waiting list.

Henrik Bengtsson (19:03:51) (in thread): > If anyone wonders what this one is about, seehttps://community-bioc.slack.com/archives/CLUJWDQF4/p1622053611008100from earlier today

2021-05-27

Andres Wokaty (08:34:48): > @Andres Wokaty has joined the channel

2021-05-28

Christian Panse (09:52:59): > @Christian Panse has joined the channel

Christian Panse (10:02:43): > ping@Hervé Pagès; The mono system does not seem to be available on the current test systems for Linux and macOS. So the rawrr BUILD and INSTALL checks fail. Is there anything I can do? Thanks.

Izaskun Mallona (10:15:17): > @Izaskun Mallona has joined the channel

Hervé Pagès (11:20:37) (in thread): > Hi@Christian Pansemono-runtimeis on the Linux builders: > > biocbuild@nebbiolo1:~$ dpkg-query -s mono-runtime > Package: mono-runtime > Status: install ok installed > Priority: optional > Section: cli-mono > Installed-Size: 98 > Maintainer: Ubuntu Developers <ubuntu-devel-discuss@lists.ubuntu.com> > Architecture: amd64 > Source: mono > Version: 6.8.0.105+dfsg-2 > Depends: libc6 (>= 2.2.5), mono-runtime-sgen (= 6.8.0.105+dfsg-2) > Description: Mono runtime - default version > Mono is a platform for running and developing applications based on the > ECMA/ISO Standards. Mono is an open source effort led by Xamarin. > Mono provides a complete CLR (Common Language Runtime) including compiler and > runtime, which can produce and execute CIL (Common Intermediate Language) > bytecode (aka assemblies), and a class library. > . > This package contains the Virtual Machine, JIT (Just-in-Time) and > AOT (Ahead-of-Time) code generator "mono-sgen". > mono-sgen executes applications for the CLI (Common Language Infrastructure). > Mono currently only supports the X86, PowerPC, ARM, S/390x, AMD64 and > MIPS architectures. > . > This package installs this architecture's default runtime version. > Original-Maintainer: Debian Mono Group <pkg-mono-group@lists.alioth.debian.org> > Homepage:[http://www.mono-project.com/](http://www.mono-project.com/) > > According to yourINSTALLfile, this should be enough for the package to “work” (mono-mcsandmono-xbuildare only needed if one needs to compilerawrr.exebut this should not be necessary since you provide this .NET executable). However it seems that ifmono-mcsandmono-xbuildare not installed,rawrr::readChromatogram()does not work properly: > > > rawrr::readChromatogram(rawfile = rawfile, type = "tic") > $times > NULL > > $intensities > NULL > > attr(,"filter") > [1] "ms" > attr(,"filename") > [1] "/home/biocbuild/.cache/R/ExperimentHub/28c00f20831d23_4590.raw" > attr(,"type") > [1] "tic" > attr(,"class") > [1] "rawrrChromatogram" > > This is something I mentioned during the review: > > Looks like on Ubuntu if I only install mono-runtime (instead ofmono-mcsandmono-xbuild) then the package installs and loads without any complain but thenrawrr::readChromatogram()silently returns a broken rawrrChromatogram object: > > > library(ExperimentHub) > eh <- ExperimentHub::ExperimentHub() > EH4547 <- normalizePath(eh[["EH4547"]]) > rawfile <- paste0(EH4547, ".raw") > if (!file.exists(rawfile)){ file.link(EH4547, rawfile) } > > library(rawrr) > x <- rawrr::readChromatogram(rawfile = rawfile, type = "tic") > x$times > # NULL > x$intensities > # NULL > > and I somehow thought that this had been addressed but apparently not. > We’ll work on getting mono on the Mac builder but we’ll wait that the exact requirement forrawrrare clarified.

Christian Panse (11:43:58) (in thread): > Thanks; the .onLoad method in zzz.R checks if mono is in the pathSys.which('mono'). This does not seem to work on the bioconductor test system, now. On my docker Ubuntu the mono-runtime seems to be sufficient. It also passed this check on the 3.13 bioc release on Linux earlier this week. For whatsoever reasons the vignette build process failed.

Christian Panse (11:51:01) (in thread): > Can you please exec(rawfile <- rawrr::sampleFilePath()) |> rawrr::readChromatogram(type='tic')on the same system. I have a suspicion that something weird is going on with the file.link.

Hervé Pagès (11:51:05) (in thread): > > This does not seem to work on the bioconductor test system, now. > It does: > > > Sys.which('mono') > mono > "/usr/bin/mono" > > > > It also passed this check on the 3.13 bioc release on Linux earlier this week. > If you look at the package landing page herehttps://bioconductor.org/packages/3.13/rawrr, the source tarball is still not available. This means thatrawrrnever passed BUILD/CHECK since we added it to the daily builds.

Christian Panse (11:52:46) (in thread): > It was passing BUILD and it showed an error on CHECK (because of a vignette problem).

Christian Panse (11:53:02) (in thread): > on Linux

Hervé Pagès (11:53:47) (in thread): > > > library(rawrr) > Package 'rawrr' version 1.0.0 using > RawFileReader reading tool. Copyright © 2016 by Thermo Fisher Scientific, Inc. All rights reserved. > > (rawfile <- rawrr::sampleFilePath()) |> rawrr::readChromatogram(type='tic') > $times > NULL > > $intensities > NULL > > attr(,"filter") > [1] "ms" > attr(,"filename") > [1] "/home/biocbuild/bbs-3.13-bioc/R/library/rawrr/extdata/sample.raw" > attr(,"type") > [1] "tic" > attr(,"class") > [1] "rawrrChromatogram" > > This is on nebbiolo1.

Christian Panse (11:54:54) (in thread): > OK; I have no idea. I need a break. Thanks a lot!

Christian Panse (12:04:29) (in thread): > correction:http://bioconductor.org/checkResults/release/bioc-LATEST/rawrr/nebbiolo1-install.htmlyou were right.

Daniela Cassol (12:12:33): > Hello all! > Ourpackageis not propagating because GO.db is not available, however, I did find any problems withGO.dbpackage. Any advice?

Christian Panse (12:18:03) (in thread): > A final test for this week. can you please exec that snippet:(rawfile <- rawrr::sampleFilePath()) |> rawrr::readChromatogram(type='xic', mass=500)it uses a different system2 call setting as the command before. Thanks again. Of note, I also adapted my~/.Renvironfromhttp://bioconductor.org/checkResults/release/bioc-LATEST/Renviron.biocin the hope of reproducing the result.

Hervé Pagès (12:35:43) (in thread): > > > library(rawrr) > Package 'rawrr' version 1.0.0 using > RawFileReader reading tool. Copyright © 2016 by Thermo Fisher Scientific, Inc. All rights reserved. > > (rawfile <- rawrr::sampleFilePath()) |> rawrr::readChromatogram(type='xic', mass=500) > xic > 10 > ms > 500 > failed to catch configfile and itol > Error accessing RAWFileReader library! - Input string was not in a correct format. > > Memory Usage: > Before 33544 kb, After 66216 kb, Extra 32672 kb > Error in file(filename, "r", encoding = encoding) : > cannot open the connection > In addition: Warning message: > In file(filename, "r", encoding = encoding) : > cannot open file '/tmp/RtmpsO3Zrx/file2043082fe672ef.R': No such file or directory > > I hope you got enough information about the problem now. With 2000 Bioconductor packages in the daily builds, this kind of troubleshooting session is not realistic. Hopefully you’ll find a way to access an Ubuntu 20.04 system where you can reproduce this. Make sure that the machine only hasmono-runtime, and never hadmono-mcsormono-xbuild. I say “never” here because installing these 2 additional components also installs dozens of dependencies, so even if you remove them, your system won’t be in the same state as if you had only installedmono-runtimeand never installedmono-mcsormono-xbuild.

Hervé Pagès (12:59:08) (in thread): > Actually, it seems that if you runsudo apt autoremoveaftersudo apt-get remove mono-mcs mono-xbuild, all the no-longer-needed deps will be removed. I just did this on my laptop and was able to reproduce therawrr::readChromatogram()issue.

Hervé Pagès (13:19:59) (in thread): > Hi@Daniela Cassol, l refactored the propagation code recently and it looks like I introduced this problem. The code “thinks” thatGO.dbis not available because it doesn’t see the Windows or Mac binaries. But we don’t distribute binary versions of annotation packages so it needs to be told to only look at whether theGO.dbsourcepackage is available. I’ll fix this. Thanks for reporting this.

Daniela Cassol (13:38:27) (in thread): > Thank you very much,@Hervé Pagès!:slightly_smiling_face:

Christian Panse (18:03:00) (in thread): > Yeah, you found it; It looks like libmono-system-data4.0-cil is the missing Linux package, but I will check more carefully next week using an Ubuntu 20.04 system. > This library is only required when extracting a chromatogram. I didn’t expect that, but it makes sense. > Your support is highly appreciated. I hope the package will be a success and all our work will pay off. thanks and happy weekend!

Hervé Pagès (18:24:40) (in thread): > Glad you found it. Have a nice weekend too!

2021-05-31

Christian Panse (10:37:02) (in thread): > Dear Hervé; Thelibmono-system-data4.0-cilpackage seems to be the only missing dependency when installing themono-runtime. I updated the INSTALL file for Linux and checked on debian:10 and ubuntu:20.04 systems (both systems behave the same). > > sudo apt-get install mono-runtime libmono-system-data4.0-cil -y > > should solve the rawrr issue on Linux. I can not wait to see rawrr source tarball on bioc. Thanks again. Christian

Henrik Bengtsson (13:48:38) (in thread): > STATS: When revdep checkingmatrixStats, 12 (7%) out of the 176 revdep packages from Bioconductor failR CMD checkwith*R_CHECK_INSTALL_DEPENDS*=trueset because they don’t declare all dependencies needed.

2021-06-04

Tobias Kockmann (08:12:52): > @Tobias Kockmann has joined the channel

Hervé Pagès (11:05:14) (in thread): > Hi Christian, 2 more things: > 1. libmono-system-data4.0-cilneeds to be listed inSystemRequirements. > 2. You need to improve error handling e.g. it’s not ok to haverawrr::readChromatogram()silently return a broken rawrrChromatogram object if some obscure dependency is missing on the user’s system. This should raise a helpful error message stating clearly what the problem is. > Thanks!

Hervé Pagès (11:15:19) (in thread): > Minor correction: what’s listed inSystemRequirementsshould beMono System.Data libraryrather thanlibmono-system-data4.0-cil. The name of the latter is specific to Ubuntu but the exact name of the package/library might be different on other Linux distributions or on Mac.

Christian Panse (13:05:00) (in thread): > Hoi Hervé; 1. done. 2. I can not wait to see my error message on the test system. I improved the error handling of the system2 call in.rawrr:::.rawrrSystem2Sourceby logging stdout and stderr and make them available from the R console. Thanks! Best wishes and happy weekend! Christian

Hervé Pagès (13:30:37) (in thread): > Sounds good. Thx

Hervé Pagès (15:37:07) (in thread): > Also please don’t forget to push these changes to theRELEASE_3_13branch of the package.

2021-06-05

Christian Panse (09:52:45) (in thread): > > 0 cp@leda:~/__checkouts/bioc/rawrr % history|grep git (git)-[RELEASE_3_13]-21-06-5|15:50:57 > 4753 git pull > 4754 git checkout RELEASE_3_13 > 4756 git merge master > 4758 git diff > 4759 git log > 4765 git add DESCRIPTION > 4766 git commit -m 'version bump 1.0.0 -> 1.0.1' > 4767 git push > 0 cp@leda:~/__checkouts/bioc/rawrr % >

Christian Panse (09:55:55) (in thread): > DONE. v1.0.1 and v1.1.6 on the test system. tnx, C

Chris Vanderaa (11:16:28): > @Chris Vanderaa has joined the channel

2021-06-07

Mike Smith (14:40:35): > Is it possible to see what version of Pandoc is used on nebbiolo2?  I’m trying to figure out why, in the header of HTML vignettes produced by the build system, imported scripts and stylesheets appear as plain text, but as base64 encoded data when I build them locally (see the attached image for an example).  Much of the style processing performed by BiocStyle is done usinggsub()on the HTML, but I get different matches on the two systems because of this discrepancy and I’m struggling to recreate the behaviour of nebbiolo2 for testing.  Different pandoc versions seems like a possible cause, although I don’t find much in the documentation to support it. - File (PNG): image.png

Andres Wokaty (14:53:40): > I’m helping Herve with nebbiolo2. This is what I see. > > biocbuild@nebbiolo2:~$ pandoc --version > pandoc 2.5 > Compiled with pandoc-types 1.17.5.4, texmath 0.11.2.2, skylighting 0.7.7 > Default user data directory: /home/biocbuild/.pandoc > Copyright (C) 2006-2018 John MacFarlane > Web:[http://pandoc.org](http://pandoc.org)This is free software; see the source for copying conditions. > There is no warranty, not even for merchantability or fitness > for a particular purpose. >

Hervé Pagès (15:53:31) (in thread): > Thanks Jennifer. Also, and not too surprisingly since we use the stock Ubuntu 20.04 pandoc on both machines, we have the exact same thing on nebbiolo1: > > biocbuild@nebbiolo1:~$ pandoc --version > pandoc 2.5 > Compiled with pandoc-types 1.17.5.4, texmath 0.11.2.2, skylighting 0.7.7 > Default user data directory: /home/biocbuild/.pandoc > Copyright (C) 2006-2018 John MacFarlane > Web:[http://pandoc.org](http://pandoc.org)This is free software; see the source for copying conditions. > There is no warranty, not even for merchantability or fitness > for a particular purpose. >

Mike Smith (16:13:04) (in thread): > Great, thanks for the info. I’m using 2.11.4 that’s distributed with RStudio. I’ll give it a whirl with the stock Ubuntu version & see if that gets me matching outputs.

2021-06-08

Christian Panse (11:07:36) (in thread): > and here there is the error messagehttp://bioconductor.org/checkResults/release/bioc-LATEST/rawrr/nebbiolo1-checksrc.htmlso the commandsudo apt-get install libmono-system-data4.0-cil -yshould solve the issue @nebbiolo1. C

Hervé Pagès (11:17:59) (in thread): > We installedlibmono-system-data4.0-cilyesterday on nebbiolo1. Also installed mono on the Mac builders.

Christian Panse (13:10:14) (in thread): > :+1:

Christian Panse (13:11:24) (in thread): > many thanks! C

2021-06-09

Oliver Voogd (21:30:33): > @Oliver Voogd has joined the channel

2021-06-14

Ludwig Geistlinger (12:04:21): > Hi, it looks as if myGSEABenchmarkeRstarted to fail with the recent Bioc 3.13 release and the update to the default BiocFileCache location:http://bioconductor.org/checkResults/devel/bioc-LATEST/GSEABenchmarkeRThe following code piece issues the BiocFileCache deprecation notice: > > hub <- ExperimentHub::ExperimentHub() > gse <- AnnotationHub::query(hub, "GSE62944") > tum <- gse[["EH1043"]] > > and seems to cause problems for digesting packages. > > Error in UseMethod("conditionMessage") : > no applicable method for 'conditionMessage' applied to an object of class "character" > Calls: loadEData ... tryCatchList -> tryCatchOne -> doTryCatch -> download.file > Execution halted > > (I also observed this for other packages depending on ExperimentHub incl.BloodGen3ModuleandsignatureSearch,http://bioconductor.org/checkResults/devel/bioc-LATEST/ExperimentHub/). Can you please advise as to how to best proceed on my end? Thanks a lot!

Lori Shepherd (12:06:13): > I reset the cache this morning and it should clear up on the next build.

Ludwig Geistlinger (12:07:19): > Alright, thanks, I’ll report back.

Federico Marini (16:18:48): > @Federico Marini has joined the channel

Federico Marini (16:20:52): > Hi, I am getting the errorC stack usage 14274688 is too close to the limitin a couple of my packages (all at once, it seems!) - only in Linux and in MacOS. See e.g.http://bioconductor.org/checkResults/devel/bioc-LATEST/pcaExplorer/merida1-buildsrc.html

Federico Marini (16:22:13): > I do not recall if that was the case, but can it be something temporary and it goes away, or is some action from my side required? I can build and check the packages with no troubles on my machines:disappointed:

Hervé Pagès (20:51:07) (in thread): > Hi Federico, make sure all your packages are up-to-date. I can reproduce this on my laptop (Ubuntu 20.10) in release and devel, but only when runningR CMD buildandnotwhen running the code in the vignette interactively. This will make debugging a little bit more complicated. Seems to be caused by something that changed in CRAN packageplotlyor one of its deps. Not sure what. For example if I replaceGRdrawDRC(drc_output)withGRdrawDRC(drc_output, plotly=FALSE)inGRmetrics’s vignette (one of the other affected packages), the problem go away. I don’t have more insight sorry.

2021-06-15

Hervé Pagès (02:08:22) (in thread): > If that helps, the problem can be reproduced by replacing all the code chunks in thepcaExplorerorflowcatchRvignette with this simple code chunk: > > library(plotly) > plot_ly(economics, x = ~pop) > > I suspect a bad interaction betweenplotlyand pandoc but it’s not clear why we only started to see this recently. Yesplotlywas updated on CRAN last week (to version 4.9.4), but I still get the error if I downgrade it to 4.9.3 or 4.9.2. One explanation could be that some other package thatplotlydepends on (or suggests) has also changed recently but I was not able to identify it yet.

Federico Marini (07:03:29) (in thread): > oh, I see. Thanks@Hervé Pagès

Federico Marini (07:04:30) (in thread): > Not a real “solution”, but would it be acceptable toeval=FALSEthese chunks? - at best only the culprit ones

Hervé Pagès (12:30:02) (in thread): > Only if it’s temporary. This plotly/pandoc issue breaks 19 packages so someone would need to go to the bottom of it.

Federico Marini (12:55:34) (in thread): > Ok, I’ll see if an issue is already open somewhere and track this further. Thanks again for the quick diagnosis!

Sehyun Oh (16:00:10): > @Sehyun Oh has joined the channel

Federico Marini (17:04:29) (in thread): > I took your example to the extreme and had a simple one chunk only - no header yaml no nothing. > Only > > library(plotly) > plot_ly(economics, x = ~pop) > > and that builds

Federico Marini (17:05:20) (in thread): > After that, some basic yaml-ing - still builds. > Error seems to be triggered when the output isBiocStyle::html_document

Hervé Pagès (17:35:37) (in thread): > I think thatBiocStyle::html_documentis needed to trigger the use of pandoc but I’m not sure. I have a feeling that pandoc doesn’t like something about the output produced byplotlyand chokes on it.

Federico Marini (18:00:19) (in thread): > as a confirmation of this hunch: the vignette knits if the output is the simple html_document

Federico Marini (18:09:05) (in thread): > I opened up a branch forflowcatchR- another package also affected by this. > Inhttps://github.com/federicomarini/flowcatchR/tree/cstack_investigationI go the “standard” html_doc output, and have added someBiocStyle::to prepend the calls that would otherwise fail.

Federico Marini (18:09:23) (in thread): > In there, there’s the plotly output that is correctly rendered

2021-06-16

Federico Marini (05:39:57) (in thread): > In case others are reading, we are investigating this inhttps://github.com/Bioconductor/BiocStyle/issues/88#issuecomment-862201912(thanks for the prompt reaction@Mike Smith!)

Hervé Pagès (12:37:16) (in thread): > Thanks@Federico Mariniand@Mike Smithfor taking care of this.

Ludwig Geistlinger (16:42:26): > Hi@Lori Shepherd: likely already noticed, but I thought I give a quick ping that mac+windows-only cache issues have persisted in release (tokay2+machv2) and devel (riesling1+merida1). > > In devel, affected packages are now seeing something like this (see eghttp://bioconductor.org/checkResults/devel/bioc-LATEST/GSEABenchmarkeR/riesling1-buildsrc.html): > > Error: processing vignette 'GSEABenchmarkeR.Rmd' failed with diagnostics: > failed to load resource > name: EH1043 > title: RNA-Sequencing and clinical data for 9246 tumor samples from The Cancer Genome Atlas > reason: Corrupt Cache: resource path > See AnnotationHub's TroubleshootingTheCache vignette section on corrupt cache > cache: D:\biocbuild\ExperimentHub_cache > potential duplicate files: > 10a05d6e27f6_2524 > 39f06c6e1264_2524 > > In release, affected packages continue to see theError in UseMethod("conditionMessage")reported above.

2021-06-17

Lori Shepherd (07:46:16) (in thread): > Thank you I’ll look into it.

Ludwig Geistlinger (08:18:58) (in thread): > Thank you

Lori Shepherd (08:24:09) (in thread): > on quick glance might have to investigate the package code more closely since the error indicates a EH number but than references different duplicate files on each platform

Ludwig Geistlinger (08:40:52) (in thread): > happy to help where I can incl debugging together or pointing out corresponding pieces of the package code

Lori Shepherd (09:07:32) (in thread): > I think I have located the bug in the hub code and will work on a fix immediately. thank you

Matthew Carlucci (09:36:00): > @Matthew Carlucci has joined the channel

Hervé Pagès (13:00:11): > <!channel>We’re modifying the build report to include raw results. For example see the new “raw results” link here:https://bioconductor.org/checkResults/3.14/bioc-LATEST/Rhdf5lib/. Following that links takes you to the collection of files that were used to generate the HTML version of the results forRhdf5lib. Making these files publicly available allows programmatic access to this information without having to scrap HTML pages. > Also all these files can be downloaded at once for faster processing: If you go onhttps://bioconductor.org/checkResults/and click on the “download” link, you’ll get a tarball (report.tgz) that contains the entire HTML report + the raw results. > Finally note that the build report also contains a summarization of the install/build/check statuses inBUILD_STATUS_DB.txt. This file is located in the top-level folder of the report e.g.https://bioconductor.org/checkResults/3.14/bioc-LATEST/BUILD_STATUS_DB.txtWe welcome any feedback.

Lori Shepherd (13:57:05): > <!channel>Please be aware. Recent changes to knitr are producing an ERROR on the build reports for many packages (software and experiment data). This is a legitimate ERROR and as a maintainer you should address the issue in any offending package as soon as possible. As an example, I fixed the Bioconductor maintained experiment data package GSE62944 this morning. The ERROR will read something like: > > Error: processing vignette 'GSE62944.Rmd' failed with diagnostics: > The 'rmarkdown' package should be installed and declared as a dependency of the 'GSE62944' package (e.g., in the 'Suggests' field of DESCRIPTION), because the latter contains vignette(s) built with the 'rmarkdown' package. Please see[https://github.com/yihui/knitr/issues/1864](https://github.com/yihui/knitr/issues/1864)for more information. > > The ERROR message knitr produces is very explicit on how to fix and is very straightforward. Please remember to always do a version bump when making changes to your package in addition to updating the Suggest field in the DESCRIPTION. Cheers.

2021-06-18

Nicholas Cooley (11:04:31): > Hi, I currently have a timeout error on tokay2 that does not appear on riesling1 on my package SynExtend. The code should be the same, so i’m a little puzzled. The only thing i can parse from the different test timings is that tokay seems to think two of my functions don’t exist in x64? But riesling does? Is this just some aberrant behavior or should i keep an eye on this?

Hervé Pagès (13:31:55): > The code is the same but the hardware is different: tokay2 is old and slow while riesling1 is a much more powerful machine. Seems likeR CMD check SynExtendis sometimes successful on tokay2 (i.e. no timeout) because the Windows binary managed to propagate (I see it on the landing page). Because the builds check many packages in parallel, depending on which other packages exactly are being checked at the same time, it could be that some daysSynExtendhas to compete for resources more than other days. > Note that the lack of computing power on Windows for the release builds is one of the reasons why we decided to run these builds only every other day. Now that we have 48 hours to complete a full build run, I see if I can decrease the number of packages that are checked simultaneously on tokay2 (24 at the moment). Maybe that will help withSynExtend’s timeout.

Hervé Pagès (13:36:48): > Hmm.. 17R CMD checktimeouts on tokay2 on the last report for the 3.13 builds:https://bioconductor.org/checkResults/3.13/bioc-LATEST/SynExtendis definitely not an isolated case.

Nicholas Cooley (14:53:29): > Ok, thanks for checking! One of my todo list items is editing my vignette and examples to make them slimmer, it sounds like that’s not something that would really help in this case?

2021-06-19

Hervé Pagès (16:35:26): > I’ve reduced the number of cpus used on tokay2. It’s always a good idea to keep the vignette and examples as slim as possible. I think it would help with the timeout on tokay2, unless there’s something else going on e.g.R CMD checkhits something and gets stuck forever. > Note that the situation will change on Windows when we stop supporting the i386 arch (32-bit). I’ve heard the CRAN folks are considering doing this for R 4.2 and so far we’ve been following their lead on this. If that’s the case this would only become effective this Fall when we set up the 3.15 builds.

2021-06-25

Ludwig Geistlinger (10:08:58) (in thread): > Hi@Lori ShepherdI just wanted to quickly follow up on this. It looks like the problem persists on the windows builders only (tokay2 in release,http://bioconductor.org/checkResults/release/bioc-LATEST/GSEABenchmarkeR/; and riesling1 in develhttp://bioconductor.org/checkResults/devel/bioc-LATEST/GSEABenchmarkeR/) … where we are still seeing theError in UseMethod("conditionMessage")… can you give it another check, or is it something that can be addressed on my end? Many thanks!

Lori Shepherd (10:19:24) (in thread): > Thanks for the reminder. I’ll try to investigate more later this afternoon after today’s reports post

Ludwig Geistlinger (10:19:45) (in thread): > Thank you

Lori Shepherd (12:35:40) (in thread): > Its odd. I can reproduce it when I do R CMD check on the build machine. But the interesting part is when I run that code manually on the builder it work fine

Ludwig Geistlinger (12:52:38) (in thread): > Agreed, I experienced similar behavior when trying it out on a Mac, it seems to be related to different behavior when running interactive or not (something likeif(interactive())seems to be involved …)

2021-06-28

Ben Story (12:13:18): > @Ben Story has joined the channel

Ben Story (12:14:22): > Not sure where I should ask this but:I have a question about including executables in a package?I’m trying to submit a package that depends on other software for part of it’s functionality. Providing a user with instructions for installing the 3rd party software on their own is easy enough but at least one function and thus parts of the vignettes error out without this 3rd part software present? Is it ok to simply include it as an executable to make things easier? What are my options in a situation like this?

Vince Carey (15:30:35): > Would you please be more specific about the executables that are needed? What are the licensing conditions?

Spencer Nystrom (21:25:42) (in thread): > One possibility if you cannot ship the exe with your package is to write anexe_is_installed()function that is user-facing. Then you can write your vignettes & examples to conditionally run only ifexe_is_installed()returnsTRUE.

2021-06-29

Ben Story (05:56:38) (in thread): > The software would be:https://github.com/cbg-ethz/SCITEso it’s all GPL-3. It’s a single compiled C++ executable but I could build on multiple systems (win/mac/linux) to help the end user.

Ben Story (18:05:12) (in thread): > This is a great idea thanks. I guess since the output of the external program is important for downstream functions I can just include post-analysis data for the other functions… Although any ideas how I can avoid the user hitting that pre-existing data if on their system the program isn’t there. I guess I could alternatively include a vignette=TRUE parameter as well.

Spencer Nystrom (19:00:14) (in thread): > I’m not exactly sure the situation you’re describing, but I used a similar approach inmemeson Bioconductor. I included several data objects that were prerendered examples of outputs so things still mostly worked. I build the package website in a docker container that has the dependency installed so the docs render, but it results in vignettes that don’t have output from Bioc builders, which I need to address, but it is an option.

2021-06-30

Hervé Pagès (01:53:32) (in thread): > Seems pretty straightforward to compile:clang++ *.cpp -o scite. In particular there’s no configure process and no reliance on external libraries. So rather than including pre-compiled executables for various architectures in your package I’d suggest including the source and compilesciteat installation-time.

Ben Story (11:21:35) (in thread): > Thanks for the reply Hervé, is there another package that I could use an example of this being done correctly or similarly?

Hervé Pagès (13:21:04) (in thread): > I think this can be achieved by placing a Makefile or configure script in the top-level folder of the package. The former being probably the preferred option. For the latter, this wouldn’t be a “real” configure script, only a small handcrafted one that contains commands for compiling scite and placing the executable somewhere under the package installation folder. > I don’t have a good example of a Bioconductor package doing this, sorry. Maybe browse/searchhttps://code.bioconductor.org/for some inspiration? You could also ask on the#package-submissionchannel.

2021-07-02

koki (05:26:47): > Has anyone else had the following error on the Daily Package Builder?https://www.bioconductor.org/checkResults/devel/bioc-LATEST/scTensor/ > > error in evaluating the argument 'x' in selecting a method for function 't': NaN is generated. Please run again or change the parameters. > > Is this the error related to knitr that Lori mentioned before?https://community-bioc.slack.com/archives/CEQ04GKEC/p1623952625100000I’m having trouble reproducing the error with thebioconductor_docker:develthat I have installed in my local machine.

Hervé Pagès (12:38:50) (in thread): > Doesn’t look like a knitr-related issue. Note that the issue is consistent on all platforms. This type of issue tends to be very reproducible but you need to make sure that: > 1. Your installation is valid and up-to-date. Check this withBiocManager::valid(). > 2. Use the same settings as the Bioconductor builders. See box “To the developers/maintainers of the scTensor package” at the top of the report forscTensor.

Ludwig Geistlinger (17:42:53) (in thread): > Hi@Lori Shepherd: did you perform any additional modifications on riesling1 (windows devel builder)? somehow the error magically disappeared and I am looking at a clean build report in devel:http://bioconductor.org/checkResults/devel/bioc-LATEST/GSEABenchmarkeR/

Lori Shepherd (20:21:57) (in thread): > I did not.

2021-07-06

Nitesh Turaga (10:48:29) (in thread): > Yes, I agree with Herve on this, this isn’t a knitr issue. It is a issue with the code itself.

2021-07-12

Ben Story (07:13:47) (in thread): > Hey so after implementing this (many thanks to Mike Smith for help) my Bioconductor reviewer for the package submission has come back withWe would prefer that you do not include external software with your package. Perhaps you can instruct users to use that software independently of yours and your package can optionally work with the output of the SCITE software? For the time being we can't accept your package with the SCITEpkg code inside of it.is this a hard and fast rule? Or do I have some wiggle room here?The external software makes the tool much more accessible to the average user (especially on Windows).

2021-07-19

Leo Lahti (16:51:28): > @Leo Lahti has joined the channel

2021-08-04

Daniela Cassol (16:52:07): > Hello! > Ourpackageis not propagating on Windows machine, even though everything seems to be okay. > Could you please advise?

Marcel Ramos Pérez (16:55:31): > Hi Daniela, it looks like theGOstatspackage binary for windows is not checking and therefore not available. We will look into it. Thanks!

Daniela Cassol (17:30:36): > Thank you very much,@Marcel Ramos Pérez!

Marcel Ramos Pérez (17:34:10): > Welcome! I’ve pushed a change. It should propagate between 24 - 48 hours.:crossed_fingers:

2021-08-05

Mikhail Dozmorov (12:48:45): > @Mikhail Dozmorov has joined the channel

2021-08-10

Hervé Pagès (13:29:18) (in thread): > Looks likeGOstatsitself is unable to propagate on Windows because ofRgraphviznot being available on this platform (currently failing on all platforms with a “the condition has length > 1” error).@Kasper D. Hansen?

Kasper D. Hansen (13:29:33): > @Kasper D. Hansen has joined the channel

2021-09-07

Davide Risso (06:24:48): > Hi all, myzinbwavepackage is failing in all platforms both in release and devel due to some tests (functions generating warnings when they should be silent). > This is the the devel report (release fails for the same reason):https://bioconductor.org/checkResults/devel/bioc-LATEST/zinbwave/nebbiolo2-checksrc.htmlAnd this is the responsible test:https://github.com/drisso/zinbwave/blob/master/tests/testthat/test_initialize.R#L90

Davide Risso (06:26:12): > I am unable to reproduce the error on any of my machines, I tried on Mac and Ubuntu (both with R 4.1 and Bioc 3.13) and with the bioconductor docker (devel)

Davide Risso (06:26:31): > What am I missing?

Davide Risso (06:27:16): > I should mention that the build started failing on Friday without any change in the code and has failed in (at least) two consecutive builds

Davide Risso (06:31:18): > Even knowing what the warning is would be helpful, but I’m not getting any warnings for that command in any of the tested systems

Hervé Pagès (13:55:36) (in thread): > Sudden failure on all platforms in release and devel is typically caused by a change in a CRAN package. Let me have a look.

Hervé Pagès (15:07:19) (in thread): > Ahhhh but also we’ve updated R from 4.1.0 to 4.1.1 on all the build machines about 10 days ago so we shouldn’t exclude a change in R 4.1.1. Are you surezinbwavestarted to fail only on Friday? We send failure notifications only on Mondays and Fridays in release and only on Wednesdays in devel. > > Here are the warnings I get on nebbiolo2 (code is taken from the “Initialization works with large epsilon” test): > > library(zinbwave) > nS <- 100 # sample size > nG <- 50 # number genes > K <- 2 # number of latent factors > W <- matrix(c(rnorm(50, mean=5, sd=.1), > rnorm(50, mean=1, sd=.1), > rnorm(10, mean=5, sd=.1), > rnorm(90, mean=1, sd=.1)), > ncol = K) > X <- matrix(1, ncol = 1, nrow = nS) > V <- matrix(1, ncol = 1, nrow = nG) > mm = zinbModel(X = X, > V = V, > W = W, > gamma_mu = matrix(1, ncol = nS), > beta_mu = matrix(5, ncol=nG)) > > my_data <- zinbSim(mm) > SE <- SummarizedExperiment(assays = list(counts=my_data$counts)) > > sf <- zinbFit(SE, K=2, epsilon=1e4) > Warning messages: > 1: In optimize(f = zinb.loglik.dispersion, Y = Y, mu = mu, logitPi = logitPi, : > NA/Inf replaced by maximum positive value > 2: In optimize(f = zinb.loglik.dispersion, Y = Y, mu = mu, logitPi = logitPi, : > NA/Inf replaced by maximum positive value > 3: In optimize(f = zinb.loglik.dispersion, Y = Y, mu = mu, logitPi = logitPi, : > NA/Inf replaced by maximum positive value > > optimize()is from thestatspackage. Maybe it has changed in R 4.1.1? > > Finally and FWIW I found 2 minor side issues with this unit test: > 1. zinbFit()is not a deterministic function. Non-deterministic functions should be avoided. > 2. The input data passed tozinbFit()is itself not generated in a deterministic way i.e. it’s generated randomly but without a call toset.seed()first. > This should be easy to address and would improve reproducibility while at the same time make things easier to troubleshoot.

Hervé Pagès (15:07:19) (in thread): > > > sessionInfo() > R version 4.1.1 (2021-08-10) > Platform: x86_64-pc-linux-gnu (64-bit) > Running under: Ubuntu 20.04.2 LTS > > Matrix products: default > BLAS: /home/biocbuild/bbs-3.14-bioc/R/lib/libRblas.so > LAPACK: /home/biocbuild/bbs-3.14-bioc/R/lib/libRlapack.so > > locale: > [1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C > [3] LC_TIME=en_GB LC_COLLATE=en_US.UTF-8 > [5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8 > [7] LC_PAPER=en_US.UTF-8 LC_NAME=C > [9] LC_ADDRESS=C LC_TELEPHONE=C > [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C > > attached base packages: > [1] stats4 stats graphics grDevices utils datasets methods > [8] base > > other attached packages: > [1] zinbwave_1.15.1 SingleCellExperiment_1.15.2 > [3] SummarizedExperiment_1.23.4 Biobase_2.53.0 > [5] GenomicRanges_1.45.0 GenomeInfoDb_1.29.8 > [7] IRanges_2.27.2 S4Vectors_0.31.3 > [9] BiocGenerics_0.39.2 MatrixGenerics_1.5.4 > [11] matrixStats_0.60.1 > > loaded via a namespace (and not attached): > [1] Rcpp_1.0.7 compiler_4.1.1 XVector_0.33.0 > [4] bitops_1.0-7 tools_4.1.1 zlibbioc_1.39.0 > [7] bit_4.0.4 annotate_1.71.0 RSQLite_2.2.8 > [10] memoise_2.0.0 lattice_0.20-44 png_0.1-7 > [13] rlang_0.4.11 Matrix_1.3-4 rstudioapi_0.13 > [16] DelayedArray_0.19.2 DBI_1.1.1 parallel_4.1.1 > [19] fastmap_1.1.0 GenomeInfoDbData_1.2.6 genefilter_1.75.1 > [22] httr_1.4.2 Biostrings_2.61.2 vctrs_0.3.8 > [25] locfit_1.5-9.4 bit64_4.0.5 grid_4.1.1 > [28] R6_2.5.1 AnnotationDbi_1.55.1 survival_3.2-13 > [31] XML_3.99-0.7 BiocParallel_1.27.5 limma_3.49.4 > [34] blob_1.2.2 edgeR_3.35.1 splines_4.1.1 > [37] KEGGREST_1.33.0 softImpute_1.4-1 xtable_1.8-4 > [40] RCurl_1.98-1.4 cachem_1.0.6 crayon_1.4.1 > > >

2021-09-08

Davide Risso (03:59:29) (in thread): > Thank Hervé! This is helpful! I will look into it when I have time, probably not today though:disappointed:

Davide Risso (04:01:01) (in thread): > As for the seed, I set the seed at the beginning of the test file, isn’t that enough to ensure reproducibility of the tests?

Davide Risso (04:03:01) (in thread): > I’m not sure that I understand your comment aboutzinbFitIt’s true that this is a non-deterministic function, but that is the nature of the problem that we are trying to solve: the function that we are trying to optimize has several local minima and depending on the initialization point we will often get different answers. Not sure how to avoid this behavior in tests other than always starting from the same initial point (which I could do but currently don’t).

Espen Riskedal (06:14:17): > @Espen Riskedal has joined the channel

Nitesh Turaga (10:33:00): > hi@Davide RissoI’ve updated the Bioconductor image (devel) to be R 4.1.1, can you try to reproduce the bug now?

Hervé Pagès (10:33:02) (in thread): > > I set the seed at the beginning of the test file, isn’t that enough to ensure reproducibility of the tests? > It’s better to set the seed at the beginning of the individual tests. That way the input that you generate randomly in a given test won’t change when you add new tests, or when the order in which the tests are run changes (e.g. because of a change intestthat), or when one manually runs that test alone for troubleshooting purposes. > > IfzinbFit()uses some kind of Monte Carlo method, one way to make it deterministic is to add aseedargument to it, with a given default value. The user would typically pass their own arbitrary seed before calling the function. Same seed should produce the same result so now the function is deterministic.

2021-09-09

Davide Risso (07:03:29) (in thread): > Hi@Nitesh Turaga, maybe I’m doing something wrong, but after runningdocker pull bioconductor/bioconductor_docker:develI still have R 4.1.0

Davide Risso (07:03:51) (in thread): > > $ docker pull bioconductor/bioconductor_docker:devel > devel: Pulling from bioconductor/bioconductor_docker > Digest: sha256:188953156dc45546eeaf624d889e9ddc11671e98721e3dae7a72b4ea118fac32 > Status: Image is up to date for bioconductor/bioconductor_docker:devel > docker.io/bioconductor/bioconductor_docker:devel >

Davide Risso (07:04:53) (in thread): > Thanks Hervé! I tried everything except the most obvious thing e.g., updating to the latest version of R! I can confirm that I can reproduce the warnings with R 4.1.1

Davide Risso (07:05:08) (in thread): > Will work on fixing this and the other issues that you raised above

Nitesh Turaga (08:39:08) (in thread): > Could you please try the following ? > > $ docker system prune > > $ docker pull bioconductor/bioconductor_docker:devel >

Nitesh Turaga (08:48:02) (in thread): > This should work.

Davide Risso (09:10:23): > In case that anyone is curious about what was the problem, I was testing the package in R 4.1.0 while the Bioc build now uses R 4.1.1.

Davide Risso (09:13:11): > This (I think) is the change that broke my package (from R 4.1.1. NEWS): > > dnbinom(20, <large>, 1) now correctly gives 0, and similar cases > are more accurate with underflow precaution. (Reported by > Francisco Vera Alcivar in PR#18072.) >

Davide Risso (09:15:28): > However, I’m still not sure exactly what’s going on and if someone could help it would be highly appreciated. In particular, this is what happens in R 4.1.0: > > > dnbinom(88, size = 1e-15, mu = 200, log = TRUE) > [1] -39.01611 >

Davide Risso (09:15:43): > In R 4.1.1: > > > dnbinom(88, size = 1e-15, mu = 200, log = TRUE) > [1] -Inf > > but > > > log(dnbinom(88, size = 1e-15, mu = 200, log = FALSE)) > [1] -39.01611 >

Davide Risso (09:17:03): > Was this the intended behavior of the change indnbinom?

Davide Risso (09:18:01) (in thread): > it worked:tada:

Davide Risso (09:18:06) (in thread): > thanks Nitesh!

Lluís Revilla (09:34:01) (in thread): > Probably not and would be a nice regression example to add to the R bug reporthttps://bugs.r-project.org

Martin Morgan (10:17:32) (in thread): > specificallyhttps://bugs.r-project.org/show_bug.cgi?id=18072. This is apparently the commithttps://github.com/wch/r-source/commit/596da3578e9ad93c5eff1d015c9193ce78de6f6a. Since there’s some impedance in reporting bugs, I sent a query to Martin Maechler directly…

Davide Risso (10:35:32) (in thread): > Thanks@Martin Morgan!

Kasper D. Hansen (10:41:24): > I don’t think in general you can expectlog(dnbinom())to be the same asdnbinom(, log=TRUE)

Alan O’C (10:48:50) (in thread): > Yeah, it’s multiplication or log addition which can differhttps://github.com/wch/r-source/blob/9be88326dd364bd4fa11f985b70c3a0d46dc421c/src/nmath/dnbinom.c#L65

Davide Risso (11:53:19) (in thread): > I feel like I should celebrate my first R bug discovery!

2021-09-15

Guido Barzaghi (13:34:35): > Goodevening everyone, I was wondering whether anyone could help me with an issue I’m trying to solve. MySingleMoleculeFootprintingpackage (devel version) throws an error upon BUILD on all platforms (see reporthere). Unfortunately, I could not reproduce the error even when using the docker container. Does anyone know how to proceed in such a situation?

Andres Wokaty (13:56:16) (in thread): > Did you try running withhttps://master.bioconductor.org/checkResults/3.14/bioc-LATEST/Renviron.bioc?

Guido Barzaghi (13:58:30) (in thread): > Yes, I initially did it with those settings on my local machine. When I couldn’t reproduce the error, I tried the docker (which should resemble the building system exactly if I understand correctly)

Andres Wokaty (14:01:00) (in thread): > I want to confirm that your local machine has R 4.1.1?

Guido Barzaghi (14:02:42) (in thread): > That’s correct

Andres Wokaty (15:20:34) (in thread): > @Guido BarzaghiI was able to replicate the error on the docker using the .Renviron.bioc file. You should check that your docker has R 4.1.1. If it doesn’t, you can do adocker pull bioconductor/bioconductor_docker:develto update your image.

Hervé Pagès (18:41:34) (in thread): > This should be reproducible on any system with Bioc devel, using R 4.1.0 or R 4.1.1, with or w/o setting the environment variables defined in Renviron.bioc. But make sure that all your packages are up-to-date by checking this withBiocManager::valid(). On my laptop: > > library(SingleMoleculeFootprintingData) > library(SingleMoleculeFootprinting) > library(BSgenome.Mmusculus.UCSC.mm10) > > CachedBamPath <- SingleMoleculeFootprintingData::NRF1pair.bam()[[1]] > CachedBaiPath <- SingleMoleculeFootprintingData::NRF1pair.bam.bai()[[1]] > > CacheDir <- ExperimentHub::getExperimentHubOption(arg = "CACHE") > file.copy(from = CachedBamPath, to = paste0(CacheDir, "/", "NRF1pair.bam")) > file.copy(from = CachedBaiPath, to = paste0(CacheDir, "/", "NRF1pair.bam.bai")) > > data.frame( > FileName = paste0(CacheDir, "/", "NRF1pair.bam"), > SampleName = "NRF1pair_DE_" > ) -> df > readr::write_delim(df, paste0(CacheDir, "/NRF1Pair_Qinput.txt"), delim = "\t") > > Qinput <- paste0(CacheDir, "/NRF1Pair_Qinput.txt") > > BaitRegions <- SingleMoleculeFootprintingData::EnrichmentRegions_mm10.rds() > BaitCaptureEfficiency <- BaitCapture(sampleSheet = Qinput, > genome = BSgenome.Mmusculus.UCSC.mm10, > baits = BaitRegions) > > Produces: > > Error in QuasR::qCount(QuasRprj, tiles, clObj = clObj) : > sequence levels in 'query' not found in alignment files: chr1_GL455991_alt, chr1_GL455992_alt, chr1_GL455993_alt, ... > > Seems related to the update of the mm10 genome by the UCSC folks. It’s now based on GRCm38.p6, was previously based on GRCm38. This adds new sequences to it, without altering the existing ones. Looks likeQuasR::qCount()expects all the sequence names present in the supplied genome to be found in the supplied BAM file. Maybe that’s too restrictive, or maybeBaitCapture()can work around this, I don’t know. > BTW please note that the portable way to build a file system path is withfile.path()e.g.file.path(CacheDir, "NRF1Pair_Qinput.txt"). Doingpaste0(CacheDir, "/NRF1Pair_Qinput.txt")is not guaranteed to work on every system.

Hervé Pagès (18:45:53) (in thread): > I’m using the latestBSgenome.Mmusculus.UCSC.mm10(v 1.4.3).

2021-09-16

Guido Barzaghi (04:58:01) (in thread): > Oh I see, that makes a lot of sense. In this case, realigning the data against the updated genome should solve the issue quite readily. Thanks a lot for the very comprehensive answer. I’ll also fix the portability issue, thanks for the advice

Henry Miller (18:34:57): > @Henry Miller has joined the channel

2021-09-17

Alan O’C (08:18:15): > Is there a way to figure out which dependencies of a package have been updated recently, beyond manually browsing bioc pages?

Charlotte Soneson (08:31:41) (in thread): > BiocPkgTools::biocBuildReport()gives you a tibble that contains, among other things, the last commit date for each package. There is also a BiocChallenge related to more or less the question you’re asking here (and the start of a package in the linked repo):https://kevinrue.github.io/BiocChallenges/articles/challenges/build_failure_tracking.html. Alsohttps://code.bioconductor.org/browse/shows you the most recent update for each package.

Alan O’C (09:14:49) (in thread): > fwiw I had two packages break recently without me pushing updates and I think the culprit is the RNG change in BiocParallelhttps://community-bioc.slack.com/archives/CLUJWDQF4/p1631804476005800

Martin Morgan (11:15:37) (in thread): > I tried to monitor breakage by parsing the status of (Imports / Depends) reverse dependencies on BiocParallel before I posted the change, and after. I used the attached script. There was only one newly reported breakage. Maybe my script is buggy, or I needed to wait another day for the effects to propagate, or maybe I should have used recursive reverse dependencies. I’ll try to get this right the next time ’round. > > What were the packages that broke? - File (R): revdeps.R

Alan O’C (11:17:26) (in thread): > scater and BASiCS, the issue wasn’t the result of the parallel code itself (as far as I can tell) but rather that the seed after bp*apply being different

Martin Morgan (11:24:59) (in thread): > thanks; the discussion on the relevant pull request (the implementation does not use the PR) seems to have settled on whetherset.seed(123); bplapply(1:3, runif)should use the global random stream (and consequently increment the stream) or whether the user needs to do something likebplapply(1:3, runif, BPPARAM = SnowParam(RNGseed = 123))and the global stream is completely ignored. The later is closer to the old behavior, the former seems much easier to use. Both are much better than the current implementation, including robustness across workers, tasks, and*Param(). Both I believe will introduce changes into packages and user scripts. My feeling is to stick with the current implementation (which does know about the global random number stream) and to always increment the stream by 1 for the reasons outlined in the comment athttps://github.com/Bioconductor/BiocParallel/pull/140#issuecomment-921627153

Martin Morgan (11:39:26) (in thread): > and the winner, in terms of what went wrong with my attempt to monitor breakage, is that theanti_join()arguments in my script are reversed — should beanti_join(current_reports, checkpoint_reports, by = c("pkg", "version"))(actually I’m not sure whether"result"should be included in theby =) > > Breaking packages (as of today) are > > > anti_join(current_reports, checkpoint_reports) |> print(n = Inf) > Joining, by = c("pkg", "version", "result") > # A tibble: 25 × 3 > pkg version result > <chr> <chr> <chr> > 1 BASiCS 2.5.4 ERROR > 2 batchelor 1.9.1 ERROR > 3 BEclear 2.9.0 ERROR > 4 BiocSingular 1.9.1 ERROR > 5 Cardinal 2.11.2 ERROR > 6 ChromSCape 1.3.21 ERROR > 7 fgsea 1.19.2 ERROR > 8 FindMyFriends 1.23.0 ERROR > 9 GDCRNATools 1.13.1 ERROR > 10 HiCBricks 1.11.0 ERROR > 11 LineagePulse 1.13.0 WARNINGS > 12 metaseqR2 1.5.12 ERROR > 13 methylGSA 1.11.0 ERROR > 14 mixOmics 6.17.26 ERROR > 15 motifbreakR 2.7.1 WARNINGS > 16 proFIA 1.19.0 WARNINGS > 17 scater 1.21.3 ERROR > 18 scDblFinder 1.7.5 WARNINGS > 19 scone 1.17.1 ERROR > 20 scPCA 1.7.1 ERROR > 21 scran 1.21.3 ERROR > 22 sesame 1.11.12 ERROR > 23 signatureSearch 1.7.3 ERROR > 24 SingleR 1.7.1 ERROR > 25 zinbwave 1.15.2 ERROR > > There are 269 reverse dependencies, so about 10% of packages break.

Alan O’C (11:45:03) (in thread): > Good to know I’m not alone:slightly_smiling_face:Thanks for the info

Alan O’C (14:29:22) (in thread): > Okay I don’t understand why incrementing the stream when no RNG is performed is preferable. It means that inserting a bplapply anywhere in a script or package changes all RNG downstream

Alan O’C (14:29:51) (in thread): > eg, > > > set.seed(42); bplapply(1, I); rnorm(1) > [[1]] > [1] 1 > > [1] 1.530677 > > set.seed(42); lapply(1, I); rnorm(1) > [[1]] > [1] 1 > > [1] 1.370958 > > vs old behaviour: > > r$> set.seed(42); bplapply(1, I); rnorm(1) > [[1]] > [1] 1 > > [1] 1.370958 > > r$> set.seed(42); lapply(1, I); rnorm(1) > [[1]] > [1] 1 > > [1] 1.370958 >

2021-09-25

Haichao Wang (07:20:10): > @Haichao Wang has joined the channel

2021-09-29

Steffen Neumann (07:24:56): > Hi, onhttps://www.bioconductor.org/developers/how-to/troubleshoot-build-report/we have instructions how to debug package build failures. It also mentions the bioconductor/bioconductor_docker:devel images. What’s missing is a paragraph how to reproduce, maybe including a snippet on doing that without the rstudio interface

Steffen Neumann (09:48:16) (in thread): > Basically, I fired up the container, useBiocManager::install("mtbls2", dependencies=TRUE)to install, well, the deps, and thendevtools::check_built("/tmp/Rtmpxyz/downloaded_packages/mtbls2_1.23.0.tar.gz")to check my package. Unfortunately I can’t reproduce the errors onhttp://bioconductor.org/checkResults/devel/data-experiment-LATEST/mtbls2/

Vince Carey (11:55:38) (in thread): > Hi@Steffen Neumann– did you followhttp://bioconductor.org/checkResults/devel/data-experiment-LATEST/Renviron.bioc… to get those specific errors in your system may take a little more configuration. I will try to get more details together soon

Vince Carey (12:00:48) (in thread): > @Hervé Pagès@Nitesh Turagaare the settings inhttp://bioconductor.org/checkResults/devel/data-experiment-LATEST/Renviron.biocbaked into the container that Steffen used?

Steffen Neumann (12:09:16) (in thread): > Probably not, as I could not reproduce the issue :-(

Vince Carey (16:16:20) (in thread): > So@Steffen Neumanncan you set those environment variables yourself?

Vince Carey (19:01:42) (in thread): > Thus: when the Renviron.bioc variables are not set, no error is thrown, but when they are: > > target "xsAnnotate" > defined "xsAnnotate" > --- function search by body --- > S4 Method plotPsSpectrum:CAMERA defined in namespace CAMERA with signature xsAnnotate has this body. > ----------- END OF FAILURE REPORT -------------- > Quitting from lines 206-223 (MTBLS2.Rmd) > Error: processing vignette 'MTBLS2.Rmd' failed with diagnostics: > 'length(x) = 16 > 1' in coercion to 'logical(1)' > --- failed re-building ‘MTBLS2.Rmd’ > > SUMMARY: processing the following files failed: > ‘MTBLS2-xcms3.Rmd’ ‘MTBLS2.Rmd’ >

Vince Carey (19:23:12) (in thread): > among other events … quite a dump file

Vince Carey (19:25:23) (in thread): > So to be completely clear (?)@Steffen Neumannyou can take the contents of the Renviron.bioc noted above, place in .Renviron in your home in the container image when running, and then you should be able to reproduce the error within the container. > > > sessionInfo() > R version 4.1.1 Patched (2021-08-27 r80822) > Platform: aarch64-apple-darwin20.5.0 (64-bit) > Running under: macOS Big Sur 11.4 > > Matrix products: default > BLAS: /Users/vincentcarey/R-4-1-dist/lib/R/lib/libRblas.dylib > LAPACK: /Users/vincentcarey/R-4-1-dist/lib/R/lib/libRlapack.dylib > > locale: > [1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8 > > attached base packages: > [1] stats graphics grDevices utils datasets methods base > > other attached packages: > [1] mtbls2_1.23.2 rmarkdown_2.11 > > loaded via a namespace (and not attached): > [1] MatrixGenerics_1.5.4 Biobase_2.53.0 > [3] vsn_3.61.2 foreach_1.5.1 > [5] assertthat_0.2.1 BiocManager_1.30.16 > [7] affy_1.71.0 stats4_4.1.1 > [9] GenomeInfoDbData_1.2.7 robustbase_0.93-9 > [11] impute_1.67.0 pillar_1.6.3 > [13] lattice_0.20-45 limma_3.49.4 ... >

Nitesh Turaga (19:32:37) (in thread): > Hi, is this on the devel image where theRenviron.biocvariables are not set?

2021-09-30

Steffen Neumann (02:52:22) (in thread): > Yes, pulled yesterday: > > REPOSITORY TAG IMAGE ID CREATED SIZE > bioconductor/bioconductor_docker devel ab0f2e24cf96 34 hours ago 3.96GB >

Nitesh Turaga (13:01:35) (in thread): > Hi, it seems3.14.24version the variables are set.

Nitesh Turaga (13:01:48) (in thread): > If you just trySys.getenv()

Steffen Neumann (13:02:44) (in thread): > I am now able to reproduce and on the issue

Nitesh Turaga (13:03:40) (in thread): > Do you see the*R**variables?

Steffen Neumann (13:04:12) (in thread): > I had to set them myself

Nitesh Turaga (13:04:40) (in thread): > How are you running the docker image?

Nitesh Turaga (13:05:00) (in thread): > Because I see them set.

Steffen Neumann (13:05:02) (in thread): > On the phone, can check later

Nitesh Turaga (13:08:33) (in thread): - File (WebM): Screenshare - 2021-09-30 7:08:05 AM.webm

Steffen Neumann (15:14:53) (in thread): > Cool, thanks for the video. So the variables are not available from the console: > > neumann@msbi-corei:/vol/R/BioC/devel/mtbls2/vignettes$ docker pull bioconductor/bioconductor_docker:devel > devel: Pulling from bioconductor/bioconductor_docker > Digest: sha256:6020dd8b7e8684cdbe458977b1ac2b16a6af4ee7574ceffe48d4259e586a9cfd > Status: Image is up to date for bioconductor/bioconductor_docker:devel > docker.io/bioconductor/bioconductor_docker:devel > sneumann@msbi-corei:/vol/R/BioC/devel/mtbls2/vignettes$ docker run -v $PWD:/bioc --rm -it -e PASSWORD=mypass -p 8787:8787 bioconductor/bioconductor_docker:devel bash -c 'echo "XX${*R_CHECK_LENGTH_1_CONDITION*}XX"' > XXXX >

Steffen Neumann (15:25:53) (in thread): > It might make sense to source/etc/environmentfrom/etc/profile.d/sourceenvironment.sh

Nitesh Turaga (15:49:09) (in thread): > it seems the*R**env variables are only available when you start R. - File (WebM): Screenshare - 2021-09-30 9:47:50 AM.webm

Nitesh Turaga (15:49:15) (in thread): > Not otherwise.

Nitesh Turaga (15:50:25) (in thread): > This is how they are introduced, > > # DEVEL: Add sys env variables to DEVEL image > # Variables in Renviron.site are made available inside of R. > # Add libsbml CFLAGS > RUN curl -O[http://bioconductor.org/checkResults/devel/bioc-LATEST/Renviron.bioc](http://bioconductor.org/checkResults/devel/bioc-LATEST/Renviron.bioc)\ > && cat Renviron.bioc | grep -o '^[^#]*' | sed 's/export //g' >>/etc/environment \ > && cat Renviron.bioc >> /usr/local/lib/R/etc/Renviron.site \ > && echo BIOCONDUCTOR_VERSION=${BIOCONDUCTOR_VERSION} >> /usr/local/lib/R/etc/Renviron.site \ > && echo BIOCONDUCTOR_DOCKER_VERSION=${BIOCONDUCTOR_DOCKER_VERSION} >> /usr/local/lib/R/etc/Renviron.site \ > && echo 'LIBSBML_CFLAGS="-I/usr/include"' >> /usr/local/lib/R/etc/Renviron.site \ > && echo 'LIBSBML_LIBS="-lsbml"' >> /usr/local/lib/R/etc/Renviron.site \ > && rm -rf Renviron.bioc >

2021-10-11

Alan Murphy (17:49:14): > @Alan Murphy has joined the channel

2021-10-12

Peter Hickey (02:40:48): > rehttps://github.com/r-lib/testthat/issues/1472: is the consensus that this is something that needs to be fixed intestthator should we be modifying our own package tests or putting in some dodgy workaround for the time being since we’re so close to release?@Constantin Ahlmann-Eltze@Aaron Lun

Hervé Pagès (16:33:19) (in thread): > We’ve changed the settings for the daily builds to: > > *R_CHECK_LENGTH_1_CONDITION*=package:*R_CHECK_PACKAGE_NAME*,abort,verbose > *R_CHECK_LENGTH_1_LOGIC2*=package:*R_CHECK_PACKAGE_NAME*,abort,verbose > > instead of: > > *R_CHECK_LENGTH_1_CONDITION*=verbose > *R_CHECK_LENGTH_1_LOGIC2*=verbose > > (seehttps://github.com/Bioconductor/BBS/commit/2060d47e82a0d5278f12445d7b2ee86fabc34cd7) so these length > 1 errors will now be reported only when they occur in the package being checked and not if they occur in one of its deps. Note that those are the settings that we were using a few months ago but we changed them in May to the more aggressive values after some discussion that happened on the#developers-forumchannel:https://community-bioc.slack.com/archives/CLUJWDQF4/p1622053611008100The new settings should be reflected on tomorrow’s build report (~Tuesday~Wednesday) and they will hopefully make thetestthatissue go away (well, the issue will remain but it will no longer affect us). - Attachment: Attachment > Hi, it looks like the Bioc build/check pipeline still doesn’t pick up all *R_CHECK_LENGTH_1_LOGIC2* bugs, i.e. the one you get from use && instead of & in expression like c(FALSE,TRUE) && c(TRUE,FALSE). > > While running revdep checks on matrixStats with these checks enabled, I run into this bug for CopywriteR but the Bioc system doesn’t pick it up, cf. https://bioconductor.org/checkResults/release/bioc-LATEST/CopywriteR/. See https://github.com/PeeperLab/CopywriteR/issues/33 for details and how to reproduce this with R CMD check.

Peter Hickey (17:40:16) (in thread): > Thanks,@Hervé Pagès! Changing those lines in my local.Renvironmeant I could reproduce the initial error in the build report and I could then confirm that making the subsequent change results in a clean build locally

Constantin Ahlmann-Eltze (17:59:00) (in thread): > Yeah, I have also now decided to comment out those offending tests as it seem to take a bit longer to get fixed than I had initially hoped:confused:

2021-10-13

Alan Murphy (12:20:42): > Hi, I’m the owner of the MungeSumstats package which I pushed changes to yesterday to sort some build errors in 3.14 nightly builds. There were no error in the builds but the windows machine did timeout:https://bioconductor.org/checkResults/3.14/bioc-LATEST/MungeSumstats/riesling1-checksrc.html. I can modify the code so the check runs quicker (likely by removing some unit tests as a quick fix) but I was looking for advice on this with the code freeze for 3.14 yesterday. I usually would see if the package builds without a timeout on a second build but wanted to get advice after this first build so I can get the changes into the 3.14 release.

Lori Shepherd (12:31:00): > I’ll leave others to recommend on thoughts about code changes. > But clarification on the freeze/release: the code freeze was for release 3.13 and we delayed a few day because of the server being down on monday. The RELEASE_3_13 branch will be frozen shortly. You can make changes to the devel (3.14) branch up to the code freeze on the 26. After that date then changes would need to be applied to the new RELEASE_3_14 (Bioc3.14) and master (devel - to be 3.15)

Lori Shepherd (12:32:22): > The only other things we ask is since the release is two weeks away , that major api changes in devel wait until after the release if you have dependent packages that may break from sudden changes

Alan Murphy (12:45:55) (in thread): > Ah okay sorry I was getting mixed up on the versions and the freeze dates. Just to clarify then, I can still make changes to devel 3.14 by pushing upstream as normal until the 26th October?

Alan Murphy (12:46:40) (in thread): > That’s perfect, I will only be changing things like unit tests to help with check run times

Alan Murphy (12:46:43) (in thread): > Thanks!

Lori Shepherd (12:52:05) (in thread): > yep

Hervé Pagès (16:54:59) (in thread): > Hi Alan, I tried to give some suggestions about this on the bioc-devel mailing list where you also asked.

Alan Murphy (17:00:43) (in thread): > Hey Hervé, thanks for your time and suggestions!

2021-10-14

Jianhai Zhang (16:57:43): > Hello, I have the error “Avoid references to external hosting platforms” in spatialHeatmap check, which downloads data from my own github repo. Is there a workaround?

Marcel Ramos Pérez (18:06:02): > Our recommendation is to either include a small example dataset in the package instead. Please seehttp://contributions.bioconductor.org/r-code.html?q=external%20data#r-cmd-check-and-bioccheckas well. - Attachment (contributions.bioconductor.org): Chapter 10 R code | Bioconductor Package Guidelines for Developers and Reviewers > Everyone has their own coding style and formats. There are however some best practice guidelines that Bioconductor reviewers will look for (see coding style). There are also some other key points,…

2021-10-15

Andres Wokaty (13:33:02): > The macOS devel builder, merida1, is down for maintenance so mac builds will not be included on today’s devel build report.

2021-10-18

Steffen Neumann (12:12:11): > Hi, any quick guesses why we see for xcms only on windows > > **** running tests for arch 'i386' ... > Running 'testthat.R' > Warning message: > In file.rename(outfile, paste0(outfile, ".fail")) : > cannot rename file 'testthat.Rout' to 'testthat.Rout.fail', reason 'The process cannot access the file because it is being used by another process' > ERROR > **** running tests for arch 'x64' ... > Running 'testthat.R' > Warning message: > In file.rename(outfile, paste0(outfile, ".fail")) : > cannot rename file 'testthat.Rout' to 'testthat.Rout.fail', reason 'The process cannot access the file because it is being used by another process' > ERROR >

Steffen Neumann (12:12:39): > This made me believe it is not our packages fault. Is it?

Hervé Pagès (12:46:16) (in thread): > We tend to see this when the code in the unit tests spans processes that hold ontestthat.Routeven after the tests have finished. This happens for example if you use parallel evaluation. It only causes problems on Windows because this OS refuses to rename a file if another process is holding on it.

Steffen Neumann (13:07:08) (in thread): > So we need to disable biocparallel ? That’ll hurt. Or add a sleep 10 and keep fingers crossed ? Can one optimise the order of tests, such that the non-parallel ones go last, and (hopefully) all leftover processes died ? Erm, finished ?

Hervé Pagès (13:28:30) (in thread): > or any combination of the above:wink:

Hervé Pagès (13:31:34) (in thread): > but my personal recommendation is to disable parallelization only on Windows, and run the tests only for the 64-bit arch

Hervé Pagès (14:56:02): > <!channel>Just a heads up that a recent change on UCSC side brokeGenomeInfoDb. > > See:https://github.com/Bioconductor/GenomeInfoDb/issues/30andhttps://github.com/Bioconductor/GenomeInfoDb/issues/31This is fixed in devel (GenomeInfoDb1.29.10) but unfortunately can’t be fixed in release (because BioC 3.13 is now frozen). > > This indirectly affects packagesBSgenome,bumphunter,ChIPpeakAnno,GenomicFeatures,karyoploteR,ODER,recount, andrtracklayer(see today’s build report). That’s only untilGenomeInfoDb1.29.10 makes it to the build machines which should happen today and be reflected on tomorrow’s report.

Michael Love (16:30:04) (in thread): > Don’t they know Bioc release schedule ?!?! > > Thanks for dealing with UCSC related issues Herve

Hervé Pagès (16:51:12) (in thread): > you’re welcome

2021-10-22

John Kerl (10:50:21): > @John Kerl has joined the channel

Hervé Pagès (15:03:40): > <!channel>Just to let you know that we’ve added RSS feeds for the Workflows, Books, and Long Tests builds, in addition to the feeds for the Software and Experiment Data builds that we’ve been generating for several years. For more information about these feeds, seehttps://bioconductor.org/developers/rss-feeds/

Hervé Pagès (17:38:33): > <!channel>Also maybe you’ve noticed all the NAs in the rightmost column of the build report for merida1 (https://bioconductor.org/checkResults/3.14/bioc-LATEST/merida1-index.html), preventing Mac binary packages from propagating. This is because the machine is no longer powerful enough to finish the builds in time. We’re aware of the issue and are currently discussing solutions. We’ll try to come up with something in the next 2 or 3 days. Thanks for your patience.

2021-10-26

Catherine Ross (11:48:42): > @Catherine Ross has joined the channel

2021-10-27

Lori Shepherd (16:57:00): > Bioconductor 3.14 is Released!! Thank you to all developers and community members for contributing to the project. See fullBioconductor 3.14 Release Announcement

2021-10-30

Kozo Nishida (21:42:37): > Hi, When do you think we will be able to pullbioconductor_docker:RELEASE_3_14? > > PS C:\Users\kozo2> docker pull bioconductor/bioconductor_docker:RELEASE_3_14 > Error response from daemon: manifest for bioconductor/bioconductor_docker:RELEASE_3_14 not found: manifest unknown: manifest unknown > > (I asked this because it is related to the update of the BioC Asia workshop materials.)

2021-10-31

Hervé Pagès (00:53:27): > I don’t know if@Nitesh Turagais on this channel. Probably better to ask on the#containerschannel.

2021-11-08

Paula Nieto García (03:18:42): > @Paula Nieto García has joined the channel

2021-11-19

Alan Murphy (04:23:28): > Hey, there doesn’t seem to have been a build report for 3.14/3.15 nightly builds yesterday. Was this expected?

Lori Shepherd (07:25:36): > There was an issue with the 3.15 report and we needed to make adjustments on our system. We should have a report for today. 3.14 is only mon, wed, and friday so there should also be a report today for 3.14

2021-11-20

Steffen Neumann (13:15:42): > Hi, I love theResults for Bioconductor software packages that depend directly on package xxxin newer build reports. Can I haz’ the same for experimental data packages ? Or was that left out on purpose ? Example:http://bioconductor.org/checkResults/3.15/bioc-LATEST/xcms/and I’d love to have that for:http://bioconductor.org/checkResults/3.15/data-experiment-LATEST/msdata/

2021-11-21

Hervé Pagès (03:40:18) (in thread): > Just to be clear, the “direct revdeps” feature doesn’t cross build boundaries. More precisely, the revdeps that you see under a software package are software packages that belong to the same build report. You don’t see revdeps that are data-exp or workflow packages there because these packages belong to other builds. It would be harder to do the “direct revdeps” thing across builds because these builds run on different schedules. > All this to say that we could easily do the “direct revdeps” thing for the data-exp reports but you would only see data-exp packages that are direct revdeps of your data-exp package. However, data-exp packages typically don’t depend on each others so you wouldn’t actually see much there. In particular you wouldn’t see software packages that are revdeps of your data-exp package, which I guess is what you are after. But as I said, this wouldn’t be easy to do.

Steffen Neumann (08:42:10) (in thread): > Thanks for the explanation, so I agree not worth it now. I still love the feature in the software builds!

Hervé Pagès (14:13:25) (in thread): > Hopefully it’s a useful feature. Glad you love it. Thanks!

2021-12-09

Quang Nguyen (12:04:25): > @Quang Nguyen has joined the channel

2021-12-14

Megha Lal (08:22:52): > @Megha Lal has left the channel

2022-01-13

Steffen Neumann (09:30:41): > Just stumbled uponhttps://developer.r-project.org/Blog/public/2021/12/07/upcoming-changes-in-r-4.2-on-windows/and has been first time I hear about it. “authors of packages using native (C, C++ or Fortran) code should read” that blog post. While it’ll be some work for mzR, in that blog it says that soon it’ll be possible to cross-compile windows packages from linux, which will relief us from having/maintaining a windows build system. Highly appreciated !

Kasper D. Hansen (10:31:06): > We used to do that (cross-compiling) WAY back in the days. You only get compilation, not testing, so I am not sure how much it will actually save for developers. A bit perhaps; time will tell.

Kasper D. Hansen (10:31:54): > That blog is super worth reading btw

Hervé Pagès (11:23:39): > Jennifer has been working on setting up a Windows builder for R UCRT. It will be added to the daily build report very soon.

Meriem Bahda (12:58:45): > @Meriem Bahda has joined the channel

Steffen Neumann (15:33:34) (in thread): > And I am in touch with Tomas Kalibera, mzR might be working without patch from day one of the BioC UCRT builder:wink:

Hervé Pagès (16:37:46) (in thread): > Hi Steffen,mzRcompiles with the new toolchain (RTools42) but the unit tests are failing:https://bioconductor.org/checkResults/3.15/ucrt-LATEST/mzR/(that’s also the case on all other platforms so probably unrelated with R UCRT/RTools42). > Note that Tomas has made a patch formzRthat gets automatically applied by R at installation-time (see**** pplying installation-time patchesline in the output ofR CMD INSTALL). Pretty cool, huh? This is temporary only i.e. the patch will eventually need to be applied for good at some point. > We’ll provide more details once palomino3 gets added to the daily builds.

2022-01-14

Steffen Neumann (05:10:35) (in thread): > Yes, I was in touch with Tomas over the last hours because that patch breaks our current development we’d love to merge into devel soon. The issues have now been resolved by setting the *R_INSTALL_TIME_PATCHES*: 'no' environment variable in the GitHub action at the correct (!) place there. Some BioC developers need to be made aware of those upcoming failures, aim must be to get rid of the need for the patches in the first place.

Steffen Neumann (05:13:00) (in thread): > The rhdf5libs by@Mike Smithet al seem to need the patch. The more dependencies, the more urgent this is to fix.

Mike Smith (05:25:40) (in thread): > I’m doing some testing on this athttps://github.com/grimbough/Rhdf5lib/tree/ucrt-testingIt’s been a bit slowed down by my own lack of understanding of the differences between rtools40 ucrt and rtools42. My initial testing with rtools40 didn’t leave a working rhdf5 when the compile time patches were applied.

Steffen Neumann (06:25:04) (in thread): > So on 4.1 it builds but you get issues in downstream package tests ? > And on 4.2 / devel there is a bigger issue as it doesn’t build in the first place ?

Steffen Neumann (06:27:01) (in thread): > I couldn’t find where you set*R_INSTALL_TIME_PATCHES*: nowhich appears inhttps://github.com/grimbough/Rhdf5lib/runs/4806537847?check_suite_focus=true#step:11:8but I had tokeepthe patches during installation of my dependencies (so they’d build in the first place) and only set*R_INSTALL_TIME_PATCHES*: noin the checks of the package in question. So don’t set in the global env of that github action. Does that give a lead ?

2022-01-18

Andres Wokaty (16:54:57): > I’ve added Palomino3 the Windows builder for R UCRT to the build report. You can see it on the long form of the report athttps://bioconductor.org/checkResults/devel/bioc-LATEST/long-report.html.

2022-01-20

Steffen Neumann (05:06:06) (in thread): > Hi, is there any way to control the application of the UCRT patches ? In our GH actions we set*R_INSTALL_TIME_PATCHES*: 'no'because Tomas’ patches were outdated. He’ll remove the patches once there is a working release. Which won’t be with the broken patches … > In a concerted effort he could remove the patches manually, but if there is an easy way to set these ENV vars for some packages upon build I’d be interested

Andres Wokaty (12:18:41) (in thread): > You might be able to set this in a.BBSoptionsfile, but I’d like@Hervé Pagèsto chime in.

Hervé Pagès (12:31:25) (in thread): > This has not been tested but you could try to add the following lines in your.BBSoptionsfile: > > INTALLprepend.win: set *R_INSTALL_TIME_PATCHES*=no&& > BUILDprepend.win: set *R_INSTALL_TIME_PATCHES*=no&& > CHECKprepend.win: set *R_INSTALL_TIME_PATCHES*=no&& > BUILDBINprepend.win: set *R_INSTALL_TIME_PATCHES*=no&& > > Try to avoid introducing spaces around=or&&. I vaguely remember them causing problems on Windows.

2022-02-28

Luke Zappia (04:43:43): > Is it possible to refresh a cached file on the build system? This failing build for****{zellkonverter}****onnebbiolo1looks to me like it might be because the example data file in the vignette has become corruptedhttps://master.bioconductor.org/checkResults/3.15/bioc-LATEST/zellkonverter/nebbiolo1-buildsrc.html. Thanks!

Alan O’C (06:32:53) (in thread): > Yeah I’ve seen the same with scater, and I saw Lori post about something similar on the devel mailing listhttps://master.bioconductor.org/checkResults/3.15/bioc-LATEST/scater/nebbiolo1-buildsrc.html

Lori Shepherd (06:45:22) (in thread): > Yes I’ll investigate later today.

2022-03-01

Lori Shepherd (07:25:12) (in thread): > We are investigating an implementation of updatingObject vs a badly downloaded file. Please be patient and we will resolve this issue over the next few days.

2022-03-03

Lukas Weber (00:51:26): > Not sure if this is the right channel, but I noticed there is a slight inconsistency betweendevtools::check()and the Bioconductor submission system /BiocCheck::BiocCheck()for new package submissions. > > If DESCRIPTION includes fields that are spread over multiple lines, e.g. forbiocViews/Imports/Suggestsetc, thendevtools::check()accepts this without a space at the end of the line starting withbiocViews:/Imports:/Suggests:. So e.g. the following works indevtools::check()without a space at the end of thebiocViews:line: > > biocViews: > Spatial, > SingleCell, > Transcriptomics > > However the Bioc submission system returns an error in this case and cancels the new submission, since it expects an extra space at the end of the line starting withbiocViews:(in DESCRIPTION). > > I’m not sure if this is due to the way the Bioc submission system reads in the submissions, or if it is an underlying standard in DESCRIPTION. It is quite a tricky error for new package submissions so thought I would mention it here.

Lori Shepherd (09:02:05) (in thread): > the biocViews issue on submission is a known issue that I’m looking into. Its actually not using BiocCheck at this point – although BiocCheck does/should check the biocViews for validity (and I think does so correctly regardless of spaces?) – the spacing issue on submission has to do with trying to precheck the file and how we are trying to accomplish it – I’m working on a fix

Lukas Weber (10:28:31) (in thread): > Ok great, thank you! Yes, I thinkBiocCheck()is working correctly when running locally for checkingbiocViews. > > (I also had a warning inBiocCheck()forImports, which I had thought was also due to the same issue. But now looking at it again, I think this was a separate issue, so I thinkBiocCheck()is working fine locally.)

Lori Shepherd (10:31:36) (in thread): > okay. thank you for the feedback

Lukas Weber (10:38:59) (in thread): > Yes, I have double-checked now. The issue is only for multi-linebiocViews:in the submission system – in which case it immediately cancels and closes the new submission, as in my first attempt at submitting my new package last night:https://github.com/Bioconductor/Contributions/issues/2542It does not affectImports:orSuggests:, and localBiocCheck::BiocCheck()is also working fine.

Lori Shepherd (10:40:34) (in thread): > yes thank you. we use ruby code in the backend and were doing a simple scan to parse of biocViews on the github DESCRIPTION – I’m working on a more robust solution now and hopefully will have updates over the next week to correct this behavior

2022-03-04

Alan O’C (17:30:02): > Is merida missing magick/imagemagic to be expected? I don’t recall seeing it beforehttp://bioconductor.org/checkResults/devel/bioc-LATEST/BASiCS/merida1-checksrc.html

Lori Shepherd (17:39:21): > We are investigating now. Yet should be resolved soon.

Alan O’C (17:43:16): > Great thanks!

2022-03-10

Lori Shepherd (09:44:27) (in thread): > this should be fixed now

Lori Shepherd (09:44:58) (in thread): > we are finalizing the investigation. this will be taken care of before the weekend. sorry for the delay.

Lori Shepherd (09:48:25): > Tentative Release 3.15 schedule has been announced. Please be mindful of deadlines:http://bioconductor.org/developers/release-schedule/

Lukas Weber (10:32:38) (in thread): > Great, thank you!

2022-03-11

Lori Shepherd (08:03:02) (in thread): > the cache’s should reset over the weekend if you are still seeing errors next week please let me know

2022-03-15

Luke Zappia (03:48:30) (in thread): > I think I’m still getting the same error as of this morninghttp://bioconductor.org/checkResults/devel/bioc-LATEST/zellkonverter/nebbiolo1-buildsrc.html:crying_cat_face:

Lori Shepherd (06:25:41) (in thread): > Ok. I’ll try and check the files. It might be an issue with the data itself. I’ll investigate more

Lori Shepherd (09:28:53) (in thread): > deceiving – it wasn’t the Zeisel Brain file that needed to be redownloaded it was the ercc-concentrations file – should be all set now

Jeanette Johnson (12:03:25): > @Jeanette Johnson has joined the channel

2022-03-17

Alan O’C (08:29:27) (in thread): > Super, thanks Lori!

2022-03-20

Sarvesh Nikumbh (21:40:54): > @Sarvesh Nikumbh has joined the channel

2022-03-21

Jianhai Zhang (14:49:07): > Hello, when building a package, I already wrote@importFrom SummarizedExperiment colData, but in the function I have to use double-colonSummarizedExperiment::colData(se) <- df.metadata. Otherwise the functioncolDatacould not be found. How to avoid the double-colon?

Jianhai Zhang (14:52:08): > By contrast,df.metadata <- colData(se)works normally without double-colon.

Peter Hickey (17:02:48): > i think you need to@importFrom SummarizedExperiment colData<-, which is the ‘setter’ whereas what you’ve imported (and what works for you currently) is the ‘getter’

Jianhai Zhang (18:28:55) (in thread): > Thanks!

2022-03-22

Steffen Neumann (04:34:54): > Hi, we are hunting down who mzR does not receive a new build.

Steffen Neumann (04:35:49) (in thread): > The important commit we need built asap is from 14.03.2022 in mzR-2.29.4 or later:https://code.bioconductor.org/browse/mzR/commits/master - Attachment (code.bioconductor.org): Bioconductor Code: Browse > Browse the content of Bioconductor software packages.

Steffen Neumann (04:36:27) (in thread): > But the devel build is stuck on mzR-2.29.3 in January:http://bioconductor.org/checkResults/3.15/bioc-LATEST/mzR/nebbiolo1-checksrc.html

Steffen Neumann (04:36:35) (in thread): > Ideas ?

Lori Shepherd (07:40:51) (in thread): > @Andres Wokatycan you check that the builder code especially regarding the pulling of packages is working correctly?

Steffen Neumann (09:40:32) (in thread): > Kurt Hornik mentioned a similar issue with rtracklayer. How good is your Germen ? > “Danke. Irgendwas ist da gerade komisch, der letzte commit fuer rtracklayer hat auch keinen rebuild getriggered (da vielleicht weil die version nicht geaendert wurde). Bin schon gespannt …”

Lori Shepherd (09:55:05) (in thread): > – I’m working on learning german but not great – but as far as my quick glance at rtracklayer – it grabbed the latest commit and is building/checking it as per the build reporthttp://bioconductor.org/checkResults/devel/bioc-LATEST/rtracklayer/as you can see the git_last_commit and the git_last_commit_date — now why users don’t get that and why it doesn’t propagate is because there was no valid version bump associated with that commit – no version bump means no propagation on our systems.

Steffen Neumann (09:58:11) (in thread): > But in the case of mzR we do have (multiple) version bumps since the 2.29.3 in January.

Lori Shepherd (09:59:47) (in thread): > lets see if it propagates on today’s report – 1.29.5 was pushed up on sunday too late to be included in yesterday (monday) report and should be reflected in today’s – if it is not than we can investigate further

Steffen Neumann (10:03:17) (in thread): > Is there any place where developers can check the push date themselves in case they forgot when they did it … ?

Lori Shepherd (10:05:15) (in thread): > I think there is an rss feed that keeps track of activity just have to find it

Lori Shepherd (10:10:58) (in thread): > http://bioconductor.org/developers/gitlog/

Hervé Pagès (13:27:12) (in thread): > Commit dates only at that link. No push dates unfortunately.

2022-03-23

Lukas Weber (01:05:29): > I have a question about R version requirements inDESCRIPTIONfor new package submissions. I saw that in some package reviews we are requiring R version >=4.2 inDependsinDESCRIPTIONfor the new package submissions. I’m wondering if this has some unintended consequences, since it means that new packages cannot be installed from GitHub (i.e. by users using a release installation of R) after they have been submitted to Bioc, in the time between now and the upcoming Bioc release. (Of course people can install in Bioc-devel, but this is probably not feasible for many users, who unlike developers may not be familiar with the Bioc-devel system.) Do we have any advice / ideas about how best to manage this? (i.e. to ensure that packages can still be installed from GitHub after submission / acceptance to Bioc?)

Lukas Weber (01:06:10): > (Sorry if wrong channel for this question)

Lukas Weber (01:29:41): > If it is currently a strict requirement to haveDepends: R (>= 4.2)then I think maybe we should reconsider this (and either just haveDepends: Ror a flexible choice of the R version number), since it creates quite a lot of friction for users if packages cannot be installed from GitHub (in a release installation of R) after submission to Bioc. I am worried this could unfortunately even lead to people deciding not to submit a package to Bioc, e.g. if they need the working GitHub install link for a paper submission or similar

Peter Hickey (02:21:31): > i’d tackle this with git branches (one for BioC submission that tracks bioc-git and one for the interim github-only version) but concede that i’m probably not the average user or developer:slightly_smiling_face:

Hervé Pagès (05:04:24): > Can we move this discussion to the#package-submissionor<#C02A1B8HUHZ>channels?

Vince Carey (05:05:22): > I’ve had this concern as well. But I do think the solution is for the developer to have a branch that is – in this case – 4.1 compatible – and be explicit with prospective installers that a ref argument must be handed to BiocManager or devtools to install in a non-devel environment. If we had a convention on this, this could likely be automated, but that is a nontrivial change to our installation support protocol, and I would say it is a relatively low priority. Failing to enforce the R version during development will lead to omissions of important restrictions and more user problems down the road, I would bet.

Lukas Weber (09:32:46): > Ok, thanks for the suggestion re branches. Yes, this will work fine for me, although I still worry this will be overly complicated for some users. Thanks, I have moved the discussion to#package-submissionnow

2022-03-24

Hervé Pagès (17:00:33): > @Kasper D. HansenMany packages failing on Windows because ofaffxparser:https://bioconductor.org/checkResults/3.15/bioc-LATEST/affxparser/Can you apply Tomas’ patch ASAP? Thanks

Henrik Bengtsson (17:11:52) (in thread): > It’s in the Bioc build queue

Kasper D. Hansen (17:55:41) (in thread): > I’m sorry, I had to deal with the crazy Swede first

Henrik Bengtsson (18:19:40) (in thread): > There was also a wicked Dane involved and we all know how they are

Hervé Pagès (19:31:42) (in thread): > Jeez, I hope everybody is fine!

2022-03-25

Jianhai Zhang (05:26:20): > Hello, I have the following example code:library(BiocParallel) ``param <- BatchtoolsParam(workers=5, cluster="slurm", template=tmpl)``register(param) ``## do work ``FUN <- function(x, y) { library(pkg); # Works } ``xx <- bplapply(1:10, FUN)FUN is an exported function in my package. FUN is working on independent workers so I have to loadpkginside FUN, but I rememberlibrary(pkg)is not allowed in the function body. Where should I placelibrary(pkg)?

Vince Carey (10:20:27): > This isn’t related to bioc-builds topic. Please pose the question on developers-forum or on the support site.

Charlotte Soneson (11:29:09): > iSEEis currently failing (since some time) only ontokay2(the windows release build,http://bioconductor.org/checkResults/release/bioc-LATEST/iSEE/tokay2-buildsrc.html). The problem seems to be in one of the vignettes, in a chunk that essentially just reads thepbmc68kdataset:https://code.bioconductor.org/browse/iSEE/blob/master/vignettes/bigdata.Rmd#L68. We can’t replicate the error on GitHub Actions, there the windows build works fine (also the Bioc devel build is fine). Trying to cross things off the list of possible causes - could there be a chance that e.g. thepbmc68kentry (fromTENxPBMCData) in the ExperimentHub cache ontokay2may be corrupted? The error seems to indicate that it’s an issue with the data: > > invalid class "SummarizedExperiment" object: > 'x@assays' is not parallel to 'x' > - Attachment (code.bioconductor.org): Bioconductor Code: Browse > Browse the content of Bioconductor software packages.

Lori Shepherd (11:44:26) (in thread): > I can reset the cache on those ids if you like EH1590 , EH1591 and EH1952 correct?

Charlotte Soneson (11:46:50) (in thread): > That would be great, thanks! Yes, EH1590-1592 seem correct.

Alex Mahmoud (14:09:54): > @Alex Mahmoud has joined the channel

2022-03-29

Charlotte Soneson (03:02:16): > Is there an issue with the release builds for the software packages? Looks like the last report is from March 23 (while the books and workflows have built as expected also in the past week).

Andres Wokaty (09:22:50) (in thread): > Thanks for bringing this up. I’ll check it out.

2022-03-31

Alan Murphy (04:12:17) (in thread): > Appears to be fixed now (thank you)!

Andres Wokaty (09:41:28) (in thread): > The win builder for release is having issues, so we’ve removed it temporarily while we investigate.

2022-04-05

Jianhai Zhang (23:51:31): > I have the warning from BiocCheck::BiocCheck: Remove set.seed usage. > > Inside my function, I need to set different seeds for multiple steps. How to resolve this warning?myfun <- function(seed.n=10) {`` set.seed(seed.n+1); step1`` set.seed(seed.n+2); step2``}

2022-04-06

Jianhai Zhang (00:28:00): > I also got the warning: The following files are over 5MB in size: ‘inst/doc/spatialHeatmap.html’“. Will it be reported in the Build Report?

Vince Carey (07:16:32) (in thread): > Please give details of the package you are talking about. What command is generating the warning?

Vince Carey (07:18:50) (in thread): > seehttps://support.bioconductor.org/p/110439/#110454

Lori Shepherd (07:19:41) (in thread): > if this is the compiled version of your vignette you should not be including the html anyways as vignettes should be generated interactively through the build system.

Marcel Ramos Pérez (12:05:03) (in thread): > https://contributions.bioconductor.org/r-code.html?q=seed#code-syntax-and-efficiency

Jianhai Zhang (14:13:40) (in thread): > My package is spatialHeatmap.BiocCheck::BiocCheckproduces this warning. Yes, the vignette is build by the system.

Henrik Bengtsson (14:37:26) (in thread): > Some comments: > > I don’t think it’s statistically sound to use +1, +2, …, i.e. you might get correlated random numbers between those sets. > > Instead, why not use just oneset.seed(seed.n)at the top? Bothstep1andstep2, will get deterministic random numbers based on that. > > In line with what Vince pointed out, if you ever find yourself having to set the random seed, make sure to undo your changes so that you do not break the random number generation stream elsewhere. > > More importantly, why do you really need to use a fixed seed? Are you 100% sure you need to do that inside a function or in a package? It’s a very rare design pattern, and it could be that there’s another approach that avoids doing it.

Lori Shepherd (14:42:26) (in thread): > that file should not be git tracked or included in git. but otherwise you should be okay with regards to your testing and the build system

Jianhai Zhang (15:45:20) (in thread): > Thank all of you for these comments! > > > Below is a detailed example. I want the multiplesample()to be random to each other as much as possible by usingseed.n <- seed + seed * (seq_len(3)-1), i.e. the randomness of onesample()is different from anothersample(). Since I assume different seeds would do it. Is the same seed enough to generate randomness among thesesample()? > > > I use mulitpleset.seed()sinceset.seed(10); sample(1:10, 5); sample(1:10, 5)is differet withset.seed(10); sample(1:10, 5); set.seed(10); sample(1:10, 5);, i.e. oneset.seed()only works for onesample(). > > > Detailed example, I want reproducible results.seed <- 10``seed.n <- seed + seed * (seq_len(3)-1)``set.seed(seed.n[1]); fil.set <- sample(fil.set, n, replace=TRUE)``set.seed(seed.n[2]); norm <- sample(norm, n, replace=TRUE)`` set.seed(seed.n[3]); dimred <- sample(dimred, n, replace=TRUE)

Jianhai Zhang (15:56:10) (in thread): > I also need fixed seed in another function, since it internally relies on reduced dimensionality (runUMAP/runTSNE from the scater package) in single cell data, which is probabilistic. I want the results repoducible, so I use twoset.seed()forrunUMAP,runTSNErespectively.

2022-04-07

Henrik Bengtsson (00:09:44) (in thread): > > I use mulitpleset.seed()sinceset.seed(10); sample(1:10, 5); sample(1:10, 5)is differet withset.seed(10); sample(1:10, 5); set.seed(10); sample(1:10, 5);, i.e. oneset.seed()only works for onesample(). > Let’s sort this one out first. Using: > > set.seed(10) > a <- sample(1:10, size = 5) > b <- sample(1:10, size = 5) > > will (i) produce the sameaandbover and over if called multiple times, and (ii)aandbare independent random samples. This is a valid and statistical sound way to fix the random seed and generate random numbers. > > There’s no need to do something like: > > set.seed(10) > a <- sample(1:10, size = 5) > set.seed(10+1) > b <- sample(1:10, size = 5) > > and it’s even incorrect and not guaranteed to be random enough. Having proper randomness is critical when doing many statistical methods - otherwise your risk getting biased and invalid results at the end (e.g. incorrect genes identified). > > When you have convinced yourself that this is the case, and using at most oneset.seed(), we can attack the remaining part.

Hervé Pagès (01:07:58): > @Jianhai ZhangPlease use proper channel (#package-submission) to further ask questions about your submission. Thanks!

Jianhai Zhang (03:14:18): > Got it.

Jianhai Zhang (03:20:39) (in thread): > Now I totally understand. Oneset.seedis enough for one R session. Thank you very much!@Henrik BengtssonI have anotherset.seedrelated question, which is related to using slurm to submit jobs to remote clusters. I will post it to the#package-submissionchannel. Thank you in advance if you could give solutions.

2022-04-14

Jianhai Zhang (21:15:05): > I got build errors:http://bioconductor.org/checkResults/3.15/bioc-LATEST/spatialHeatmap/palomino3-buildsrc.html. It there anything to do on my part?

2022-04-15

Jianhai Zhang (14:02:09) (in thread): > Everything is fine now.

Nicole Ortogero (15:18:52): > @Nicole Ortogero has joined the channel

2022-04-22

Lluís Revilla (09:58:05): > Hi, is there an up to date document with the build schedule for software, annotation, workflows and books? > > The developer troubleshooting of the website mentions that (software) packages are checked every day (http://bioconductor.org/developers/how-to/troubleshoot-build-report/). > I remember seeing somewhere on the mailing list that currently software packages are checked only on Mondays, Wednesdays and Fridays (but perhaps were only annotation packages or workflow packages/vignettes).

Lluís Revilla (09:58:33) (in thread): > I’ve looked at the BBS github page (https://github.com/Bioconductor/BBS/), where it links a file (https://docs.google.com/document/d/1Ubp7Tcxr1I1HQ8Xrp9f73P40YQ0qhWQ_hSmHs05Ln_M/edit#heading=h.r7sorafgdpnf) which is outdated (servers names do not match with those currently used).

Lori Shepherd (10:03:10) (in thread): > http://bioconductor.org/checkResults/lists the schedule

Lluís Revilla (10:05:03) (in thread): > Thanks!!

Marcel Ramos Pérez (10:09:53) (in thread): > Perhaps we should add those blurbs at the top of each build report page:thinking_face:

Lluís Revilla (10:14:07) (in thread): > It might help. There is already a link to the troubleshoot-build-report webpage there (at least for software packages), perhaps is easier to find the for all types of builds schedule there (and indexed by searchers)

Lori Shepherd (10:15:36) (in thread): > I can add the link and info to the troubleshooting so its listed somewhere there

Christopher Eeles (14:12:40): > @Christopher Eeles has joined the channel

Christopher Eeles (14:14:33): > Seems like something has gone wrong with today’s build on Windows Server 2022. I have a cascade of package errors originating ingenefudue to missingamapCRAN package andCoreGxdue to missingbackportspackage. Both CRAN packages passed their CRAN checks today. > > Can someone confirm this is a build system issue and I don’t need to take action?

2022-04-23

Lori Shepherd (11:20:03): > Looks like it cleared up today so you should be all set

2022-04-26

Nicholas Cooley (10:16:30): > Hi, not sure if this is the right place for this question, but: when we specify3.15in the bioc version before3.15is released, BiocManager installs packages from Devel? > > i.e. running:FROM r-base:4.2.0``ENV BIOC_VERSION "3.15"``RUN Rscript -e "BiocManager::install(version = '$BIOC_VERSION') ; BiocManager::install(c('DECIPHER', 'SynExtend', 'rtracklayer'))"in a docker container yesterday would pull everything from bioc devel, but doing it tomorrow after release would pull everything from release? It looks that way from the constructed container, I just want to make sure.

Hervé Pagès (14:50:27) (in thread): > BiocManager::install(version = '3.15')will always pull everything from the 3.15 repos. That’s not going to change with the release. What is going to happen tomorrow is that 3.15 will be officially released butBiocManager::install(version = '3.15')will still pull everything from the 3.15 repos which will then be considered the “release repos” (and the various “release” symlinks on the website that are currently pointing to 3.14 will be changed to point to 3.15).

2022-04-27

Nicholas Cooley (09:17:29) (in thread): > ok, thanks for the explanation!

Lori Shepherd (17:32:28): > Bioconductor 3.15 is released! See full release announcement:https://bioconductor.org/news/bioc_3_15_release/

Lori Shepherd (17:32:42) (in thread): > Thanks to all developers and community members for contributing to the project!

Kozo Nishida (19:37:59): > Thank you for the releasing, but I noticed that the vignette in my package wasn’t updated.https://www.bioconductor.org/packages/release/bioc/html/transomics2cytoscape.htmlWhat do I need to do to update it? - Attachment (Bioconductor): transomics2cytoscape > transomics2cytoscape generates a file for 3D transomics visualization by providing input that specifies the IDs of multiple KEGG pathway layers, their corresponding Z-axis heights, and an input that represents the edges between the pathway layers. The edges are used, for example, to describe the relationships between kinase on a pathway and enzyme on another pathway. This package automates creation of a transomics network as shown in the figure in Yugi.2014 (https://doi.org/10.1016/j.celrep.2014.07.021) using Cytoscape automation (https://doi.org/10.1186/s13059-019-1758-4).

Lori Shepherd (19:58:47): > Can you expand on what you mean by wasn’t updated?Did you push a change that you don’t see?

Kozo Nishida (20:09:36): > I’m sorry I didn’t make it clear enough. > Yes, I pushed a change that I don’t see. I pushed it before the repository froze.

Lori Shepherd (20:41:40): > I’ll investigate and get back to you.

2022-04-28

Lori Shepherd (08:05:28) (in thread): > The last change I see is from last Thur April 21? If the commit was after this in the git log it appears it was never push togit.bioconductor.org > > commit ebaf488bbd223e1ee20e62a13f0fd186453efa94 (upstream/master) > Author: Kozo Nishida <kozo.nishida@gmail.com> > Date: Thu Apr 21 22:09:49 2022 +0900 > > Remove warnings from R CMD check (#26) > > * Add tibble to Imports > > * Include ec2reaction output tsv in extdata/usecase1 > > * Update the filenames for ec2reaction function > > * Remove RCy3:::.defaultBaseUrl > > The above is the last commit togit.bioconductor.org

Lori Shepherd (08:06:03) (in thread): > I don’t see any changes to a vignette committed?

Kozo Nishida (08:24:09) (in thread): > Oh, I’m sorry. You are right. > It’s my fault. I hadn’t checked thoroughly. > Certainly the content of that commit is reflectedhere.

Lori Shepherd (08:25:40) (in thread): > let us know if there are any other questions. cheers

Kozo Nishida (08:42:37): > The above was due to my lack of confirmation. sorry… > I have another question. > > After the repository froze, I noticed the following minor corrections are needed in my package: (and I pushed it to RELEASE_3_15 branch) > * Forgot to update NEWS > * Forgot to update the “last updated” date of the beginning of my vignette > * Minor mistakes in vignette (one letter difference, etc.) > If I wait for a while, will it (==push to RELEASE_3_15 branch after the release) be reflected in my package release someday? > (By the way, I haven’t updated z in my package version x.y.z for that change.)

Lori Shepherd (08:46:45): > yes. You can now update both the RELEASE_3_15 branch and the master (devel == future 3.16 branch) at any time. You have to do a z version bump for that version of the package to propagate to users as we don’t propagate just on updates it needs a z bump; but on the next build with a valid version bump those changes would be reflected in the available package and on the landing pages. Please see the main checkResults page that has the schedule of rebuildshttp://bioconductor.org/checkResults/

Kozo Nishida (08:51:47) (in thread): > I got it. Thanks for the information!

2022-05-02

Peter Hickey (19:08:12): > I’m trying to understand whyscRNAseqis not available fromBiocManager::install()in devel (Intel macOS) > > > BiocManager::install('scRNAseq') > 'getOption("repos")' replaces Bioconductor standard repositories, see '?repositories' > for details > > replacement repositories: > CRAN:[https://cran.rstudio.com/](https://cran.rstudio.com/)Bioconductor version 3.16 (BiocManager 1.30.17), R 4.2.0 (2022-04-22) > Installing package(s) 'scRNAseq' > Warning: unable to access index for repository[https://bioconductor.org/packages/3.16/data/annotation/bin/macosx/contrib/4.2](https://bioconductor.org/packages/3.16/data/annotation/bin/macosx/contrib/4.2): > cannot open URL '[https://bioconductor.org/packages/3.16/data/annotation/bin/macosx/contrib/4.2/PACKAGES](https://bioconductor.org/packages/3.16/data/annotation/bin/macosx/contrib/4.2/PACKAGES)' > Warning: unable to access index for repository[https://bioconductor.org/packages/3.16/workflows/bin/macosx/contrib/4.2](https://bioconductor.org/packages/3.16/workflows/bin/macosx/contrib/4.2): > cannot open URL '[https://bioconductor.org/packages/3.16/workflows/bin/macosx/contrib/4.2/PACKAGES](https://bioconductor.org/packages/3.16/workflows/bin/macosx/contrib/4.2/PACKAGES)' > Warning: unable to access index for repository[https://bioconductor.org/packages/3.16/books/bin/macosx/contrib/4.2](https://bioconductor.org/packages/3.16/books/bin/macosx/contrib/4.2): > cannot open URL '[https://bioconductor.org/packages/3.16/books/bin/macosx/contrib/4.2/PACKAGES](https://bioconductor.org/packages/3.16/books/bin/macosx/contrib/4.2/PACKAGES)' > Warning message: > package 'scRNAseq' is not available for Bioconductor version '3.16' > > A version of this package for your version of R might be available elsewhere, > see the ideas at[https://cran.r-project.org/doc/manuals/r-patched/R-admin.html#Installing-packages](https://cran.r-project.org/doc/manuals/r-patched/R-admin.html#Installing-packages) >

Peter Hickey (19:09:35) (in thread): > Is the Warning about an issue withinst/NEWS.Rdin the check meaning the package isn’t propagating?https://bioconductor.org/checkResults/devel/data-experiment-LATEST/scRNAseq/nebbiolo2-checksrc.html

Lori Shepherd (20:37:34) (in thread): > Yes. We’ve been having some issues with the devel builders/propagation. We think it should be remedied on tht next build. Sorry for the inconveniences

2022-05-12

Michael Lawrence (11:01:22): > @Michael Lawrence has joined the channel

Michael Lawrence (11:02:51): > I noticed that biovizBase is not building on Windows due to a missing “backports” package. Is that just missing from the Windows build machine? Thanks for any help.

Andres Wokaty (12:16:18) (in thread): > I’ll take a look.

Andres Wokaty (13:40:47) (in thread): > It looks like the error cleared up on this report. I saw the error but am not sure what caused it. I did update Palomino3’s R yesterday.

Kasper D. Hansen (16:53:17): > Is it intentional that we do not provide binaries for macOS M1. Super confusing to people in my class

Hervé Pagès (18:23:16) (in thread): > We’re considering building M1 binaries at some point but we don’t have the hardware yet. Note that this is a recurrent question: seehttps://support.bioconductor.org/p/9137290/#9137342. It’s also in our release announcement:https://bioconductor.org/news/bioc_3_15_release/

2022-05-13

Vince Carey (05:31:01) (in thread): > The lack of a rack-mountable M1 device is delaying our engagement with M1 binaries. While Simon Urbanek has been using mac minis for CRAN binary production, this would not be straightforward for us.

Kasper D. Hansen (09:54:06) (in thread): > This is going to be a big problem for users. One thing is the fortran compiler; that’s easy. Another thing is other dependencies which is needed for some packages (I think).

Kasper D. Hansen (09:54:27) (in thread): > I am happy to put a Mac Mini in my office if that would help

Michael Lawrence (13:24:24) (in thread): > Thanks a lot for looking into this! Nice to see that it cleared up.

Andres Wokaty (14:56:39) (in thread): > We started to talk about the hardware and hosting today, but it will time to get things running. It does help to know that this is wanted.

2022-05-14

Hervé Pagès (01:52:03) (in thread): > @Kasper D. HansenNot sure why is this a big problem for users when they can simply use R Intel 64-bit binary from CRAN:thinking_face:

Kasper D. Hansen (09:24:19) (in thread): > Because when you do to CRAN, it looks like you should install the M1 build. And because CRAN distributes binary packages you’ll also hear from a lot of people that the M1 build is what you want

Hervé Pagès (11:48:54) (in thread): > Don’t you get a bunch of warnings that some repositories can’t be accessed when you doBiocManager::install()with the M1 R binary from CRAN? And then, if you proceed, packages get installed/compiled from source instead of binaries. So if you’re an end user who is used to install binaries, from the very start you cannot miss the fact that something is not right.

Kasper D. Hansen (11:52:36) (in thread): > You get warnings but it downloads binaries as far as I recall. I can check on my M1 in a bit

Hervé Pagès (13:42:16) (in thread): > Well, it will download binaries from CRAN packages but not for BioC packages because we don’t make them.

2022-05-15

Vince Carey (07:02:16) (in thread): > Here’s the standard event for me when I use BiocManager with M1-based R: > > > BiocManager::install("parody", force=TRUE) > Bioconductor version 3.15 (BiocManager 1.30.17), R 4.2.0 RC (2022-04-15 r82206) > Installing package(s) 'parody' > Warning: unable to access index for repository[https://bioconductor.org/packages/3.15/bioc/bin/macosx/big-sur-arm64/contrib/4.2](https://bioconductor.org/packages/3.15/bioc/bin/macosx/big-sur-arm64/contrib/4.2): > cannot open URL '[https://bioconductor.org/packages/3.15/bioc/bin/macosx/big-sur-arm64/contrib/4.2/PACKAGES](https://bioconductor.org/packages/3.15/bioc/bin/macosx/big-sur-arm64/contrib/4.2/PACKAGES)' > Warning: unable to access index for repository[https://bioconductor.org/packages/3.15/data/annotation/bin/macosx/big-sur-arm64/contrib/4.2](https://bioconductor.org/packages/3.15/data/annotation/bin/macosx/big-sur-arm64/contrib/4.2): > cannot open URL '[https://bioconductor.org/packages/3.15/data/annotation/bin/macosx/big-sur-arm64/contrib/4.2/PACKAGES](https://bioconductor.org/packages/3.15/data/annotation/bin/macosx/big-sur-arm64/contrib/4.2/PACKAGES)' > Warning: unable to access index for repository[https://bioconductor.org/packages/3.15/data/experiment/bin/macosx/big-sur-arm64/contrib/4.2](https://bioconductor.org/packages/3.15/data/experiment/bin/macosx/big-sur-arm64/contrib/4.2): > cannot open URL '[https://bioconductor.org/packages/3.15/data/experiment/bin/macosx/big-sur-arm64/contrib/4.2/PACKAGES](https://bioconductor.org/packages/3.15/data/experiment/bin/macosx/big-sur-arm64/contrib/4.2/PACKAGES)' > Warning: unable to access index for repository[https://bioconductor.org/packages/3.15/workflows/bin/macosx/big-sur-arm64/contrib/4.2](https://bioconductor.org/packages/3.15/workflows/bin/macosx/big-sur-arm64/contrib/4.2): > cannot open URL '[https://bioconductor.org/packages/3.15/workflows/bin/macosx/big-sur-arm64/contrib/4.2/PACKAGES](https://bioconductor.org/packages/3.15/workflows/bin/macosx/big-sur-arm64/contrib/4.2/PACKAGES)' > Warning: unable to access index for repository[https://bioconductor.org/packages/3.15/books/bin/macosx/big-sur-arm64/contrib/4.2](https://bioconductor.org/packages/3.15/books/bin/macosx/big-sur-arm64/contrib/4.2): > cannot open URL '[https://bioconductor.org/packages/3.15/books/bin/macosx/big-sur-arm64/contrib/4.2/PACKAGES](https://bioconductor.org/packages/3.15/books/bin/macosx/big-sur-arm64/contrib/4.2/PACKAGES)' > installing the source package 'parody' > > as noted it just builds from source subsequently. The warning stream is unpleasant.

Vince Carey (07:04:16) (in thread): > Would it be worthwhile to tweak BiocManager to respond to this situation a little differently?

2022-05-16

Kasper D. Hansen (09:26:37) (in thread): > I think so

2022-05-31

Dan Bunis (14:26:32): > Not sure if this belongs here or somewhere else(?), but it seems like the ranking badge on package pages are currently in reverse order. I see it for multiple packages, so probably universal… Maybe a ‘-’ got removed or introduced somewhere recently:shrug:? Example:Biobaserelease version shows “2135/2140” but should probably be “6/2140”.

Lori Shepherd (14:29:56): > thanks I’ll investigate

2022-06-01

Lori Shepherd (14:17:59): > We were having trouble with the stat generation last week which I think threw off the rankings. They have been regenerated and I believe everything should be corrected now

2022-06-02

Mike Smith (15:31:31): > What version(s) of pandoc are installed on the builders? I’m most interested in whichever one makes the rendered vignettes available from the package landing pages. I’m seeing some different formatting with BiocStyle on my local machine and think it might be pandoc related.

Andres Wokaty (15:55:45) (in thread): > both nebbiolo (linux) machines use > > pandoc 2.5 > Compiled with pandoc-types 1.17.5.4, texmath 0.11.2.2, skylighting 0.7.7 >

Hervé Pagès (16:21:13) (in thread): > Note that this is the Pandoc that belongs to Ubuntu 20.04, so lagging quite behind the latest version (pandoc 2.18, released in April 2022).

Marcel Ramos Pérez (16:28:13) (in thread): > Maybe this info should be added here?https://bioconductor.org/checkResults/3.16/bioc-LATEST/nebbiolo2-NodeInfo.html

Mike Smith (16:46:21) (in thread): > Thanks a lot for the info. I suspect I’m using a newer version that ships with Rstudio, but will confirm in the morning.

2022-06-17

George Odette (17:25:33): > @George Odette has joined the channel

2022-06-27

Henrik Bengtsson (19:20:29): > Hi, normally I try to reach out to maintainers when I spot it issue from revdep checks, but too busy to that right now. However, I want to suggest that Bioconductor runsR CMD checkon vignettes once in a while, because some packages fail their vignette checks, often because they don’t declare all the packages that their vignette require (in addition to their package code). > > For example, autonomics, CNVgears, eegc, metaboliteIDmapping, and netboost, do not declare ****BiocStyle**** as a dependency (typically inSuggests:), giving errors such as: > > * checking re-building of vignette outputs ... ERROR > Error(s) in re-building vignettes: > ... > --- re-building 'using_autonomics.Rmd' using rmarkdown > Error: processing vignette 'using_autonomics.Rmd' failed with diagnostics: > there is no package called 'BiocStyle' > > This is withR CMD check --as-cran ..., but, if I recall correctly, can be picked up byR CMD checkby setting*R_CHECK_SUGGESTS_ONLY*=true(or something similar).

2022-07-07

Alan O’C (12:24:45): > Any hot tips on debugging MacOS arm errors? Github actions doesn’t support it yet so I think the BBS is the only feedback I’ve got

Andres Wokaty (13:00:08): > What is the error? I’ve added our M1 to the devel software report; however, we’ve still got a lot of errors and I had a big misconfiguration with gfortran that won’t be resolved until tomorrow’s report.

Kasper D. Hansen (20:39:32): > I have an M1 and I can probably help if you post details, including github link

2022-07-08

Athena Chen (12:00:18): > @Athena Chen has joined the channel

Alan O’C (17:04:07): > If there’s a tonne of errors overall still I’ll wait until it’s a bit more stable, but I may post again if errors persist. Was just a bit flummoxed at a wall of red errors

2022-07-21

Alan Murphy (10:30:53): > Hey! I’m the maintainer of the MungeSumstats(https://github.com/neurogenomics/MungeSumstats) package which is currently failing checks and I was looking for a bit of help debugging - The package is failing nightly checks for Bioc 3.16 on linux and mac:https://bioconductor.org/checkResults/3.16/bioc-LATEST/MungeSumstats/nebbiolo2-checksrc.htmlbut passing on windows. The issue is, I suggest the installation of a package newly added to Bioc 3.16 - SNPlocs.Hsapiens.dbSNP155.GRCh38: > > Package suggested but not available: 'SNPlocs.Hsapiens.dbSNP155.GRCh38' > > The suggested packages are required for a complete check. > > It has been 3 days since I pushed the changes so I thought if the issue was going to resolve itself, it would have by now. Anyone know what’s doing on?

Hervé Pagès (14:44:37) (in thread): > The problem on Linux disappeared on today’s report. > The installation log forSNPlocs.Hsapiens.dbSNP155.GRCh38on lconway has: > > 2: In download.file(url, destfile, method, mode = "wb", ...) : > downloaded length 2677029821 != reported length 6166721421 > 3: In download.file(url, destfile, method, mode = "wb", ...) : > URL '[https://bioconductor.org/packages/3.16/data/annotation/src/contrib/SNPlocs.Hsapiens.dbSNP155.GRCh38_0.99.22.tar.gz](https://bioconductor.org/packages/3.16/data/annotation/src/contrib/SNPlocs.Hsapiens.dbSNP155.GRCh38_0.99.22.tar.gz)': Timeout of 300 seconds was reached > Warning in download.packages(pkgs, destdir = tmpd, available = available, : > download of package 'SNPlocs.Hsapiens.dbSNP155.GRCh38' failed > > @Andres WokatyWe should probably increaseR_DEFAULT_INTERNET_TIMEOUTto 600 (from 300 right now) to give enough times to the build machines to install big packages like those.R_DEFAULT_INTERNET_TIMEOUTis set here:https://github.com/Bioconductor/BBS/blob/32ca60423ae328f68ccf5441bafe8f15f646edb6/3.16/Renviron.bioc#L40Thanks!

Alan Murphy (15:54:40) (in thread): > Thanks Hervé!

2022-07-22

Alan Murphy (13:29:01) (in thread): > Increasing the limit would be great, I see it failed on Iconway for the same reason today

Andres Wokaty (13:31:28) (in thread): > I increased the limits; however, it might have run before I put them on lconway. I will take a look at it later. If it fails again tomorrow, maybe we need to increase it more.

Alan Murphy (13:31:58) (in thread): > Great thanks!

2022-08-04

Alan O’C (06:30:38): > Looks like there might be a cache issue forZeiselBrainData()again, this time just on palominohttp://bioconductor.org/checkResults/devel/bioc-LATEST/miQC/palomino4-buildsrc.html

Lori Shepherd (07:49:58) (in thread): > I reset the cache for that resource. hopefully should clear up on the next build

2022-09-05

Alan O’C (06:50:27): > Are there build badges for books? Near as I can tell it should be a matter of just adding an extra looping variable to the Rakefile task but I’m not 100% sure, I get 404ed on the master.bioconductor link therehttps://github.com/Bioconductor/bioconductor.org_archive/blob/11e3056bf06b9bc2a6e7647551949fb2a92ced91/Rakefile#L506-L536

Lori Shepherd (09:55:55): > Currently the books do not have badges or traditional landing pages.

2022-09-07

Lori Shepherd (16:37:59): > Bioconductor 3.16 Release Schedule and Deadlines

2022-09-16

Luke Zappia (02:45:26): > Are there any current issues build issues? The last build report for****{zellkonverter}****is from Monday for v1.6.4 but the landing page is showing v1.6.5 which I pushed this week to fix a bughttp://bioconductor.org/checkResults/release/bioc-LATEST/zellkonverter/.

Lori Shepherd (05:52:12): > I’ll check the landing page generation and get back to you thanks.

Lori Shepherd (06:57:56): > @Andres WokatyIt looks like the builds didn’t run on Wed. can we investigate why and if we are on track to have a release report today?

Andres Wokaty (08:49:51): > lconway wasn’t able to finish the other day. we should be able to get a report today.

Andres Wokaty (10:21:19): > I posted the build report but noticed there was an issue with palomino4, which I had to remove.

Lori Shepherd (10:25:06): > those are devel – release didn’t post wed?

Andres Wokaty (10:36:56): > I will look at release soon. i thought maybe I had introduced some problems with related to the changes I have been making on devel recently.

Andres Wokaty (12:18:45): > Release ran without issue today:https://bioconductor.org/checkResults/3.15/bioc-LATEST/

Alan Murphy (13:15:57): > Hi, I’m the maintainer of the EWCE package and have noticed it is failing on the mac machine for release 3.15 -https://bioconductor.org/checkResults/3.15/bioc-LATEST/EWCE/merida1-buildsrc.html. I haven’t updated the release version since it went live so I’m not sure why it’s failing now? I waited till the build updated today but there still seems to be an error showing

Lori Shepherd (13:17:29) (in thread): > I can reset the cache for the eh resource. my guess is the download got stopped or truncated in an incomplete state. I’ll jump on now and we can see if it clears up next week

Alan Murphy (13:17:47) (in thread): > Thank you!

2022-09-21

Alan Murphy (06:17:32) (in thread): > That reset worked, thanks!

2022-09-26

Henrik Bengtsson (19:25:08): > I pushedilluminaio 0.39.1 the other day and it’s now tested and available, cf.https://bioconductor.org/packages/devel/bioc/html/illuminaio.html. However, the newNEWS.mdis not recognized on the package page, which still shows the old, no longer existing,NEWSfile. Is this a hiccup in the package page generator?

Andres Wokaty (20:20:44) (in thread): > I’m looking into this.

Andres Wokaty (21:16:35) (in thread): > Unless I’m missing something, it looks like the content of the new file. There is some delay between the creation of the report and propagation (when is when theNEWSis created). There’s also caching the website.

Kasper D. Hansen (21:33:36) (in thread): > Yeah Henrik the site I look at has the commenst form 0.39.1. But damn the start is confusing referring to completely different version numbers

Lori Shepherd (21:39:56) (in thread): > Yes as@Andres Wokatystated there is normally a delay of a few hours from when the builds propagate to the landing page generation. And as mentioned there is page caching so clearing a browser of history/cookies might also help to see the refreshed page

2022-09-27

Henrik Bengtsson (00:01:06) (in thread): > Ah, it has nothing to do with delays or caches. It was that the newNEWS.mdhad a few malformed headers and two versions that were way too large. It’s been fixed in the next Bioc push, cf.https://github.com/HenrikBengtsson/illuminaio/commit/d730d1a354e1508802b72b1be0e13b6725f89762. > > Sorry for the noise, and thanks for the prompt responses. > > PS. Ctrl-Shift-R force reloads the web browser ignore any cached content, including cached CSS

Andres Wokaty (13:35:14): > Mac ARM64 binaries are now available for devel. You can view the build report athttps://bioconductor.org/checkResults/3.16/bioc-mac-arm64-LATEST/. > > To make use of these binaries > > * InstallR-4.2.1-arm64.pkgfrom CRAN athttps://cran.r-project.org/bin/macosx/* Install**** BiocManager ****as usual***** Switch to BioC devel with**BiocManager::install(version="devel")* Use BiocManager::install() as usual. > > Note that BiocManager::install() will automatically pick the new arm64 binaries so you should no longer need Xcode.

Jennifer Holmes (16:15:04): > @Jennifer Holmes has joined the channel

2022-09-28

Hans-Rudolf Hotz (01:27:18): > @Hans-Rudolf Hotz has joined the channel

2022-09-29

Alan O’C (08:38:36): > A lot of packages havesubscript out of boundserrors on macOS arm64 eg,http://bioconductor.org/checkResults/devel/bioc-LATEST/a4Classif/lconway-buildsrc.htmlbut all my packages have it also

Lori Shepherd (08:46:31): > we are aware and looking into the issue. thank you for reporting

Alan O’C (08:55:40): > I figured, thanks! Was half posting here as a placeholder as I’m sure others will see it too

Andres Wokaty (11:33:47): > Just a note that lconway is the intel mac, kjohnson is the arm64 mac and its build report athttp://bioconductor.org/checkResults/3.16/bioc-mac-arm64-LATEST/is posted separately since it’s running on a different schedule.

2022-10-06

Dario Righelli (05:51:22): > Hi all, I don’t know if this is the right place to post this but I’m having a building error from one of my packages and I’m having troubles in fixing this newly introduced bug. > My question is: is there a deadline for fixing an error on a package? > Thanks! > Dario

Lori Shepherd (07:39:02) (in thread): > Depends – if we know you are actively working on trying to remedy the issue we will not deprecate your package – The release is fast approaching and once we freeze a release we will not allow back porting of any changes – so October 17 th is the freeze deadline for the current RELEASE_3_15 branch or it would be frozen broken – you would have all release cycle to fix the current master (soon to be release 3.16) and subsequent devel (tentative release 3.17)

Dario Righelli (07:40:32) (in thread): > Thanks Lori, I’m working on it and I’m pretty confident I’ll be able to fix it for the freezing deadline:sweat_smile:

2022-10-10

Dario Righelli (05:19:17) (in thread): > Back on this thread, I’m trying to restore an old commit in my package git. I’ve done that in local and on my github, but I’m not able to reflect these changes in the upstream, it rejects my push. Is there anything I have to do for doing that?

2022-10-11

Dario Righelli (04:25:47) (in thread): > Sorry to ping you@Lori Shepherd, any feedback on this?

Lori Shepherd (07:31:19) (in thread): > it was a holiday yesterday for us – please be patient when asking core team for help – its a busy time for us – > How was the old commit reestablished? Bioconductor does not allow force push. Please make sure that all version numbers are corrected depending on the situation too.

Lori Shepherd (07:31:43) (in thread): > It would be helpful to also show commands and output rather than just “it rejects my push”

Dario Righelli (09:38:02) (in thread): > So sorry about my ping, I didn’t want to put any pressure on that, I wasn’t sure you’ve been notified about my message. > > You’re right, I’m just force pushing an old commit, the one that I’m trying to push is 5 months old. > These are the commands I used. > > > git reset --hard b0426a98d06af142fd259bcc5c0cb76158c41159 > > At this point I updated the version and the date, committing it. > Then pushed the changes on my github > > > git push -f > > git push upstream master > To git.bioconductor.org:packages/DEScan2 > ! [rejected] master -> master (non-fast-forward) > error: failed to push some refs to 'git.bioconductor.org:packages/DEScan2' > hint: Updates were rejected because the tip of your current branch is behind > hint: its remote counterpart. Integrate the remote changes (e.g. > hint: 'git pull ...') before pushing again. > hint: See the 'Note about fast-forwards' in 'git push --help' for details. > > But if I pull, I restore the not working version… > So I tried … > > > git push -f upstream master > Enumerating objects: 15, done. > Counting objects: 100% (15/15), done. > Delta compression using up to 4 threads > Compressing objects: 100% (9/9), done. > Writing objects: 100% (9/9), 946 bytes | 22.00 KiB/s, done. > Total 9 (delta 6), reused 0 (delta 0), pack-reused 0 > remote: FATAL: + refs/heads/master packages/DEScan2 drighelli DENIED by fallthru > remote: error: hook declined to update refs/heads/master > To git.bioconductor.org:packages/DEScan2 > ! [remote rejected] master -> master (hook declined) > error: failed to push some refs to 'git.bioconductor.org:packages/DEScan2' > > Thanks for your answer and sorry again for tagging you.

Lori Shepherd (09:43:38) (in thread): > is there any way to just apply these manually to the branch instead of restoring and force pushing? In a force push scenario currently Bioconductor would have to reset your repo on git which is really not ideal and runs the risk of losing branches and commits — in order for Bioconductor to reset your github version repo would have to have the RELEASE_X_Y branches synced with the Bioconductor ones and no extra development branches so we could reclone it over — (we are re-evaluating this whole process and hopefully will find something less detrimental and extreme)

Dario Righelli (09:45:13) (in thread): > ok, so maybe it’s better for me to restore the not working version and to manually change it to the previous one, because my previous version is exactly the last release, but I need the other branches at the moment:smile:

Dario Righelli (09:46:03) (in thread): > Thanks again, I’ll try to do that today!:smile:

Lori Shepherd (09:46:51) (in thread): > thank you

Dario Righelli (10:04:59) (in thread): > ok seems it worked! I’ll wait for the server auto-test! > Thanks again Lori and sorry again

Lori Shepherd (10:06:00) (in thread): > no worries. glad to help. thank you for doing it that way instead of the force / reset

Dario Righelli (10:16:41) (in thread): > no problem!:wink:

2022-10-17

Jianhai Zhang (21:23:00): > I documented my function “covis” (a method for SVG class) below. However, I got two warnings. Can anyone give solutions? The same problem exists for the function “spatial_hm”. The repo to reproduce warnings:https://github.com/jianhaizhang/spatialHeatmap,roxygen2::roxygenize('spatialHeatmap'); devtools::build('spatialHeatmap', vignettes = FALSE); rcmdcheck::rcmdcheck('spatialHeatmap_2.1.3.tar.gz');Documentation of covis.R:#' @name covis``#' @rdname covis,SVG-method``#' @docType methodsWarnings: > Undocumented S4 methods: > generic ‘covis’ and siglist ‘SVG’ > generic ‘spatial_hm’ and siglist ‘SVG’ > > checking Rd sections … > Objects in without in documentation object ‘covis’: > ‘4method{covis}{SVG}’ > > Objects in without in documentation object ‘spatial_hm’: > ‘4method{spatial_hm}{SVG}’

2022-10-18

Hervé Pagès (01:22:44) (in thread): > Sorry I thought this was going to be a question about our daily builds (I misinterpreted “questions related to package checking”), but I see now that it’s not. This is really a question about package development and has nothing to do with our daily builds so I’ll reiterate my advice to ask on the bioc-devel mailing list. Thx

Hervé Pagès (01:24:55) (in thread): > If for whatever reasons you’re trying to avoid bioc-devel (but I see no reason why), you can try the#developers-forumchannel in this Slack.

Jianhai Zhang (12:29:34) (in thread): > Ok, thanks.

2022-10-22

Jianhai Zhang (21:09:33): > Hello, there is a cache folder~/.cache/shmfor the vignettes in my package spatialHeatmap (https://github.com/jianhaizhang/spatialHeatmap). Since this package is updated, the old files in this folder could cause errors in the daily build. Can any administrator delete this cache folder (rm -fr ~/.cache/shm) for me? Thanks. > In addition, is the latest dev version ofExperssionAtlaspackage (http://bioconductor.org/checkResults/devel/bioc-LATEST/ExpressionAtlas/) installed on Bioc dev? If not, there will be downloading errors in my package (dev) during the daily check?

2022-10-23

Henrik Bengtsson (20:32:25): > Does Bioconductor check system ever runR CMD checkon the vignettes? Looking at the check pages it doesn’t look like it, i.e. they report: > > * checking running R code from vignettes ... SKIPPED > * checking re-building of vignette outputs ... SKIPPED > > Could it be that do you do it before new Bioc releases? > > PS. The reason why I’m asking is that when I run full revdep checks on my CRAN packages, it’s not unusual that I end up with Bioconductor packages failing their vignette checks, e.g. due to not loading/declaring all needed package.

2022-10-24

Hervé Pagès (02:09:00) (in thread): > Vignettes are already validated byR CMD buildso we don’t build them again duringR CMD checkto not waste CPU cycles (we use the--no-vignettesflag for that). I didn’t know thatR CMD checkreports it if a package does not declare a package used in one of its vignettes.:thinking_face:

Henrik Bengtsson (03:30:49) (in thread): > I see. Unfortunately, I don’t think you can pick up non-declared dependencies withR CMD build. AFAIU, it’ll build the package with whatever packages you have installed at build time. In contrast, vanillaR CMD checkpicks things up such as: > > * checking for unstated dependencies in vignettes ... NOTE > '::' or ':::' import not declared from: 'dplyr' > 'library' or 'require' call not declared from: 'tidyverse' > > When I runrevdepcheck, which checks in a sandbox where only declared dependencies are available (because ofR CMD check --as-cran+ maybe some more*R_CHECK**settings), I’ll get: > > * checking re-building of vignette outputs ... ERROR > Error(s) in re-building vignettes: > --- re-building 'DEWSeq.Rmd' using rmarkdown > Loading required package: DEWSeq > Loading required package: R.utils > Loading required package: R.oo > Loading required package: R.methodsS3 > R.methodsS3 v1.8.2 (2022-06-13 22:00:14 UTC) successfully loaded. See ?R.methodsS3 for help. > R.oo v1.25.0 (2022-06-12 02:20:02 UTC) successfully loaded. See ?R.oo for help. > > Attaching package: 'R.oo' > ... > Quitting from lines 506-507 (DEWSeq.Rmd) > Error: processing vignette 'DEWSeq.Rmd' failed with diagnostics: > could not find function "read_tsv" > --- failed re-building 'DEWSeq.Rmd' > > The is because theDEWSeqvignette (<https://code.bioconductor.org/browse/DEWSeq/blob/master/vignettes/DEWSeq.Rmd?branch=master&file=vignettes/DEWSeq.Rmd) relies onrequire(tidyverse)to attachreadrso thatread_tsvis defined. But,tidyverseis not specified as a dependency, which meansrequire(tidyverse)will silently fail (returning FALSE), andreadrwill therefore also never be attached. > > Other examples I often run into are Bioconductor packages forgetting to declareBiocStyle(e.g. underSuggests:), e.g. > > * checking re-building of vignette outputs ... ERROR > Error(s) in re-building vignettes: > ... > --- re-building 'using_autonomics.Rmd' using rmarkdown > Error: processing vignette 'using_autonomics.Rmd' failed with diagnostics: > there is no package called 'BiocStyle' > --- failed re-building 'using_autonomics.Rmd' > > PS. It’s not a rare thing that Bioconductor packages forget to declare dependencies in the vignettes - it’s rather common actually.

Vince Carey (05:11:04) (in thread): > I’ve been advocating that we collect longitudinal data on check and other QC-related events in the bioc software ecosystem over time, so that this “not a rare thing” can be assessed precisely. Do you have an approach to this@Henrik Bengtsson? It seems natural to think about a centralized database with some information derived from check logs for each package.

Vince Carey (06:28:00) (in thread): > Example, using some stuff in vjcitn/BiocBuildTools to analyze 3.15 as frozen: > > > errs |> filter(grepl("there is no package called", errors)) > package errors > 1 MIGSA Error in library(mGSZ) : there is no package called ‘mGSZ’ > 2 ppiStats there is no package called ‘ScISI’ > 3 RNASeqR there is no package called 'RNASeqRData' > > and I think this speaks to a value of having “error codes”. (This output does not reflect the state of the official build system. It is a result of an experiment in which I try to understand how to operate on the ecosystem as a whole. It ran afoul of some shortcomings in latex resources, caches that fill disks, and so on. But there is tooling to produce a database of adverse events in a Bioc build and I hope we can make this more accessible and useful.)

Lori Shepherd (07:23:25) (in thread): > it looks like sat build did not occur; we are investigating. As long as the new version of ExpressionAtlas builds/checks without error it would be available for the builders on the next build run we propagate. > As far as the cache is concerned – how do you implement your cache? A good caching system will not only store files but update when an updated version of the file occurs

Kasper D. Hansen (09:23:32): > As Herve said, we don’t do this. Am I wrong in saying thatR CMD checkdidn’t use to do anything special for vignettes. That seems to have changed though. However, given the cost of processing vignettes in Bioc (for some packages this is a substantial part of the checking) and given the total cost of processing everything in the build system, this is not a small ask. Something we should do though.

Kasper D. Hansen (09:23:53): > Doescheckdo anything else than flag missing dependencies?

Alan O’C (10:05:12) (in thread): > Maybe BiocCheck should parse declared dependencies and vignette dependencies and setdiff them

Lluís Revilla (10:07:24) (in thread): > I’m not sure if parsing will work with dependencies such as those of the box package or those like tidyverse and readr.

Alan O’C (10:10:17) (in thread): > R CMD check classes that as an undeclared dependency, too

Alan O’C (10:10:38) (in thread): > (the tidyverse example, don’t know what you mean about box)

Lluís Revilla (10:12:04) (in thread): > Box is a way to use files as python does: via moduleshttps://cran.r-project.org/web/packages/box/vignettes/box.html

Andres Wokaty (11:46:56) (in thread): > I removed/.cache/shmon our devel builders.

Hervé Pagès (12:33:18) (in thread): > Yes, building the vignettes again duringR CMD checkwould introduce a high cost in terms of additional build time but the Linux builders could easily take it. However the Windows and Mac builders could not: they’re not powerful enough and they already have the extra burden of building package binaries. So maybe we could compromise by dropping--no-vignetteson Linux only?

Hervé Pagès (12:35:47) (in thread): > It seems that the additional feedback that we’ll get from this is platform independent anyways.

Henrik Bengtsson (13:17:00) (in thread): > It’s all happening in <https://github.com/wch/r-source/blob/trunk/src/library/tools/R/check.R>

Henrik Bengtsson (13:18:40) (in thread): > It could be that you could do something more granular using_R_CHECK_nnnenvironment variables, cf. <https://github.com/wch/r-source/blob/trunk/src/library/tools/R/check.R>

Jianhai Zhang (14:02:31) (in thread): > I just saved the files with BiocFileCache, not considering version info. What do you suggest?

Lori Shepherd (14:10:47) (in thread): > I would need more information on why the old files could cause errors – were they replaced with new ones? was there an update somewhere?

Henrik Bengtsson (14:44:04) (in thread): > > Do you have an approach to this@Henrik Bengtsson? > Sorry, I don’t. But, I think it would be great if Bioconductor could use its leverage and working with CRAN so that theR CMD checkframework, which is implemented in thetoolspackage, could be opened up with a generic API. This way you could imagine running with custom check flavors, e.g.--flavor=cran,--flavor=bioconductor, etc. This framework would allow you to prototype and explore new checks, while still being run viaR CMD check. You can also imagine being able to control whether you want to escalate or deescalate certain issues. I can imagine Bioconductor adds its own rules and some of them will probably be useful for CRAN too, and then it’ll be easy for CRAN to incorporate them into their set of checks. It’s been a while, but I jotted down my thoughts on this in <https://github.com/HenrikBengtsson/Wishlist-for-R/issues/16>. > > With such a system, you’d be able to add plugins that outputs the check results in some parsable format and then run statistics on them.

Hervé Pagès (15:24:59) (in thread): > That file has hundreds of occurrences of_R_CHECK_somethingand I’ve no idea where these things are documented. Do you have one in mind in particular that would address the problem of undeclared vignettes deps?

Jianhai Zhang (15:25:14) (in thread): > The criteria for filtering RNA-seq data are changed in the new vignette, so the old filtered data saved in the cache are not compatible with the new vignette.

Henrik Bengtsson (15:56:19) (in thread): > Sorry, I don’t; I always run withR CMD check --as-cranand only rarely have to tweak to get more checks to run. I suspect several of these are “internal”, so the code is the documentation. This another reason for my proposal to makeR CMD checkmore plug’n’play (https://community-bioc.slack.com/archives/CEQ04GKEC/p1666637044722169?thread_ts=1666571545.883419&cid=CEQ04GKEC) - Attachment: Attachment > > Do you have an approach to this @Henrik Bengtsson? > Sorry, I don’t. But, I think it would be great if Bioconductor could use its leverage and working with CRAN so that the R CMD check framework, which is implemented in the tools package, could be opened up with a generic API. This way you could imagine running with custom check flavors, e.g. --flavor=cran, --flavor=bioconductor, etc. This framework would allow you to prototype and explore new checks, while still being run via R CMD check. You can also imagine being able to control whether you want to escalate or deescalate certain issues. I can imagine Bioconductor adds its own rules and some of them will probably be useful for CRAN too, and then it’ll be easy for CRAN to incorporate them into their set of checks. It’s been a while, but I jotted down my thoughts on this in <https://github.com/HenrikBengtsson/Wishlist-for-R/issues/16>. > > With such a system, you’d be able to add plugins that outputs the check results in some parsable format and then run statistics on them.

Kasper D. Hansen (15:58:07) (in thread): > Seems like runningR CMD checkon the vignettes but only on linux will get us there

Hervé Pagès (17:48:55) (in thread): > Agreed. Sounds like the easiest way to go for now. Will also keep the CHECK results easy to reproduce.

Hervé Pagès (18:01:46) (in thread): > Done:https://github.com/Bioconductor/BBS/commit/086e161b91de67a3332cb81b0dabe103a021e6cc@Andres WokatyJen can you please deploy? Thx

2022-10-25

Jianhai Zhang (14:04:36) (in thread): > Did palomino4/Windows server check packages yesterday? My package has the same error as before cache was deleted.http://bioconductor.org/checkResults/devel/bioc-LATEST/spatialHeatmap/palomino4-buildsrc.html

Jianhai Zhang (14:09:10) (in thread): > Or the cache on Windows is under a different path instead of~/.cache/shmso that the old one was not deleted.

Andres Wokaty (14:16:04) (in thread): > It was under a different path but I did remove it

Jianhai Zhang (14:17:55) (in thread): > Did palomino4/Windows server check packages yesterday?

Andres Wokaty (14:18:30) (in thread): > The builds ran yesterday on Palomino4

2022-10-26

Hervé Pagès (00:53:48) (in thread): > Thx@Andres WokatyToday’s BBS run just finished on nebbiolo2. New total running time on nebbiolo2: 10h instead of 8h. Not too bad. Note that a few packages are now timing out during CHECK, which was to be expected (R CMD checktime limit is still 40min). See build report in about 10 hours (around 11 am EST).

Jianhai Zhang (15:12:46) (in thread): > If warnings and errors are not fixed by Friday October 28, what will happen?

Lori Shepherd (15:16:12) (in thread): > it depends on how long the package has been failing and if it is failing across all platforms and if a maintainer is responsive or not – packages failing for an extended period of time across all platforms with no effort to fix will be candidates for deprecation – if maintainers are actively trying to fix the package this will not happen; however failing packages that continue to fail will then have to be fixed on both the RELEASE_3_16 branch (once its created next Tuesday) and the master (devel 3.17) branch – we advise changes to be in by this friday even tho you can commit up to the branching date because it will allow the changes to be accurately tested on the daily builder

Jianhai Zhang (15:18:12) (in thread): > Got it. I will try to fixed the errors by Fri. Now I have difficulty to reproduce the errors.

2022-10-27

Jianhai Zhang (14:12:37) (in thread): > I increased the version to 2.3.2 on Oct 26 10 am PDT, why is the version bump failing in yesterday build? See version bumpatgit@git.bioconductor.org:packages/spatialHeatmap.git

Andres Wokaty (14:36:49) (in thread): > The builds start at 2pm ET. You have to push to Bioconductor before that time in order to get it on the next day’s build report.

Jianhai Zhang (14:51:07) (in thread): > 10 am PDT is 1 pm ET.

Andres Wokaty (14:53:20) (in thread): > Yes, I know. That’s when the commit happened. When did you push?

Jianhai Zhang (14:53:47) (in thread): > Yes, you can check git@git.bioconductor.org:packages/spatialHeatmap.git

Andres Wokaty (15:04:02) (in thread): > So in order for the build system to pick it up, you had to push it to Bioconductor with the version bump before 2pm ET. The commit and push are different.

Jianhai Zhang (15:11:26) (in thread): > The commit and push, including version bump was 1PM ET. See the log from git@git.bioconductor.org:packages/spatialHeatmap.git - File (PNG): Screenshot from 2022-10-27 12-10-18.png

Andres Wokaty (15:13:27) (in thread): > This log only shows the commit.

Jianhai Zhang (15:14:43) (in thread): > I cloned first (gitclonegit@git.bioconductor.org:packages/spatialHeatmap.git), then showed the log. It means the version bump commit was pushed and recorded in remote git repo.

Jianhai Zhang (15:19:08) (in thread): - File (PNG): Screenshot from 2022-10-27 12-18-39.png

Andres Wokaty (15:20:16) (in thread): > git logshows only your commits. It doesn’t show when you didgit pushto your Bioconductor repository.

Jianhai Zhang (15:20:58) (in thread): > How can I check the time of each push?

Andres Wokaty (15:21:40) (in thread): > That’s a good question. I’m looking for that too.

Andres Wokaty (15:27:15) (in thread): > It seems that git doesn’t track that information.

Jianhai Zhang (15:27:51) (in thread): > Ok. Now the bump is in remote bioc repo, so version bump will take effect in today build?

Andres Wokaty (15:29:46) (in thread): > I confirmed the build system has the new commits. it will show up in the report tomorrow.

Jianhai Zhang (15:30:11) (in thread): > ok.

2022-10-28

Brian Schilder (08:30:05): > @Brian Schilder has joined the channel

2022-11-02

Nicholas Cooley (13:26:51): > Hi, I have an error in check on Nebbiolo that is confusing me a little; > > The error most likely occurred in: > > > base::assign(".ptime", proc.time(), pos = "CheckExEnv") > > ### Name: Generic > > ### Title: Model for predicting PID based on k-mer statistics > > ### Aliases: Generic > > ### Keywords: datasets > > > > ### **** Examples > > > > data(Generic) > free(): invalid next size (fast) > Aborted (core dumped) > > Check has a problem loading one my RData files, but the problem doesn’t materialize on the other platforms. Is this a one-off occurrence? I can spin up a docker and start testing this if not.

Henrik Bengtsson (13:45:47) (in thread): > > free(): invalid next size (fast) > Aborted (core dumped) > > FWIW, as a clue, this “Aborted” messageprobablycomes from outside of R. When R decides to abort (== “crashes beyond no return”), it outputs “R is aborting now …” (interactive) or “An irrecoverable exception occurred. R is aborting now …” (non-interactive), cf.https://github.com/wch/r-source/commit/9b72106db761bc0bfd0c50aa641658dc29caa098

Nicholas Cooley (13:52:16) (in thread): > Thanks! That was my assumption based on the fact that it passed check on the other two platforms. But if this is something I can avoid with a simple change I’d like to do that, if I can.

Aidan Lakshman (13:57:03): > @Aidan Lakshman has joined the channel

Henrik Bengtsson (14:00:26) (in thread): > Sorry for not being clear; it could still mean there’s a bug in the code (C/C++ code?) of the package or one of its dependencies. I just wanted to say it’s less likely it’s happening in the base R code

Aidan Lakshman (14:02:19) (in thread): > there’s C code in the package, but I haven’t seen any errors like this with local testing

Lori Shepherd (14:14:26): > Bioconductor 3.16 is Released. Thanks to all developers and community members for contributing to the project! Please see the full release announcement:https://bioconductor.org/news/bioc_3_16_release/

Henrik Bengtsson (14:26:48) (in thread): > Unless when they’re obvious, core dumps can often be “random”, i.e. not perfectly reproducible. If there’s multi-threadingor forked parallelization involved, then you can get such random behavior from lurking bugs. I would approach it’s a real bug, that happens rarely. PS/ FYI: What “the package” is, is unknown to the audience here

Aidan Lakshman (14:30:47) (in thread): > sorry–this is an error we got in the build report for the SynExtend package duringR CMD CHECKon Linux. No multi threading or forked parallelization is involved in the package to my knowledge. I did find one variable I forgot to deallocate in a C method; I’m wondering if the cause was a very slight memory leak that led to a crash that’s hard to reproduce locally

Henrik Bengtsson (15:23:07) (in thread): > Got it; the error log discussed is at <https://bioconductor.org/checkResults/devel/bioc-LATEST/SynExtend/nebbiolo1-checksrc.html>. There are Unix tools to check for memory leaks and coding mistakes, e.g. valgrind and “sanitizers” ASan and UBsan. > > To avoid setting them up locally, which can take some efforts, you can try to check your package through the awesome R-hub service (<https://docs.r-hub.io/>). When you’ve got your email address registered, runrhub::check()from your package source folder and run it through theirlinux-x86_64-rocker-gcc-sanplatform, and possibly alsoubuntu-rchk, e.g. > > > rhub::check() > ── Choose build platform > > 1: Debian Linux, R-devel, clang, ISO-8859-15 locale (debian-clang-devel) > 2: Debian Linux, R-devel, GCC (debian-gcc-devel) > 3: Debian Linux, R-devel, GCC, no long double (debian-gcc-devel-nold) > 4: Debian Linux, R-patched, GCC (debian-gcc-patched) > 5: Debian Linux, R-release, GCC (debian-gcc-release) > 6: Fedora Linux, R-devel, clang, gfortran (fedora-clang-devel) > 7: Fedora Linux, R-devel, GCC (fedora-gcc-devel) > 8: Debian Linux, R-devel, GCC ASAN/UBSAN (linux-x86_64-rocker-gcc-san) > 9: macOS 10.13.6 High Sierra, R-release, brew (macos-highsierra-release) > 10: macOS 10.13.6 High Sierra, R-release, CRAN's setup (macos-highsierra-release-cran) > 11: Oracle Solaris 10, x86, 32 bit, R-release (solaris-x86-patched) > 12: Oracle Solaris 10, x86, 32 bit, R release, Oracle Developer Studio 12.6 (solaris-x86-patched-ods) > 13: Ubuntu Linux 20.04.1 LTS, R-devel, GCC (ubuntu-gcc-devel) > 14: Ubuntu Linux 20.04.1 LTS, R-release, GCC (ubuntu-gcc-release) > 15: Ubuntu Linux 20.04.1 LTS, R-devel with rchk (ubuntu-rchk) > 16: Windows Server 2022, R-devel, 64 bit (windows-x86_64-devel) > 17: Windows Server 2022, R-oldrel, 32/64 bit (windows-x86_64-oldrel) > 18: Windows Server 2022, R-patched, 32/64 bit (windows-x86_64-patched) > 19: Windows Server 2022, R-release, 32/64 bit (windows-x86_64-release) > > Selection: > > You can also try to check your package via valgrind locally, which requires installing valgrind on Linux first: > > rcmdcheck::rcmdcheck(build_args = "--no-build-vignettes", args = c("--use-valgrind") > > Note that this is really slow, so it’ll take a long time. If you’re package is on GitHub, you can use GitHub Actions toR CMD checkyour package via valgrind, e.g. <https://github.com/HenrikBengtsson/illuminaio/blob/develop/.github/workflows/R-CMD-check-valgrind.yaml> and <https://github.com/HenrikBengtsson/illuminaio/actions/runs/3372640458>.

Henrik Bengtsson (15:26:18) (in thread): > BTW, if you haven’t noticed yet, the compiler warning reported during install (<https://bioconductor.org/checkResults/devel/bioc-LATEST/SynExtend/nebbiolo1-checksrc.html>) is a strong suspect for memory leaks. I would guess that valgrind and/or ASan/UBsan will detect this and hopefully give an even more informative error message: > > * installing to library '/home/biocbuild/bbs-3.17-bioc/R/library' > * installing **source** package 'SynExtend' ... > **** using staged installation > **** libs > gcc -I"/home/biocbuild/bbs-3.17-bioc/R/include" -DNDEBUG -I/usr/local/include -fpic -g -O2 -Wall -c CDend.c -o CDend.o > CDend.c: In function 'scorePMs': > CDend.c:521:18: warning: 'idxchange' may be used uninitialized in this function [-Wmaybe-uninitialized] > 521 | memcpy(longPm[idxchange], longPm[longl-i-1], lh); > | ^ > gcc -I"/home/biocbuild/bbs-3.17-bioc/R/include" -DNDEBUG -I/usr/local/include -fpic -g -O2 -Wall -c R_init_synextend.c -o R_init_synextend.o > gcc -I"/home/biocbuild/bbs-3.17-bioc/R/include" -DNDEBUG -I/usr/local/include -fpic -g -O2 -Wall -c SEutils.c -o SEutils.o > gcc -I"/home/biocbuild/bbs-3.17-bioc/R/include" -DNDEBUG -I/usr/local/include -fpic -g -O2 -Wall -c calcMIR2C.c -o calcMIR2C.o > gcc -shared -L/home/biocbuild/bbs-3.17-bioc/R/lib -L/usr/local/lib -o SynExtend.so CDend.o R_init_synextend.o SEutils.o calcMIR2C.o -L/home/biocbuild/bbs-3.17-bioc/R/lib -lR >

Aidan Lakshman (15:32:19) (in thread): > ah that’s true–there shouldn’t be any case where that value isn’t initialized before accession, but I’ll add in a default value just in case.valgrindetc. is on my list, but it’s kind of a pain to set up since they’re not supported on OSX–if the latest updates don’t fix the package then I’ll try that instead. Thanks!

Henrik Bengtsson (15:39:57) (in thread): > Whenever you use C/C++/Fortran in your R package, I don’t think there’s a shortcut to not ever testing with valgrind/ASan/UBSan. Writing bug free C/C++/Fortran code is near impossible without the right tools. Without full tests, all you can do is hope for the best - but almost always it’ll come back and bite your or your users. So, I really recommend R-hub - it’s a fabulous service to the R community. Likewise, getting set up on GitHub and testing via GitHub Actionsisworth it, and easy when you’re up and running. I cannot imagine R package development without either of these services - together withR CMD check -as-cran, they save me so much struggle and time

Aidan Lakshman (15:41:37) (in thread): > That’s superhelpful, I really appreciate it—before this push I was just using a little bit of C, butI’mrealizing thatI’llprobably have to set up a more robust testing pipeline now thatI’mwriting mostly C code

2022-11-03

Ying Chen (01:36:56): > @Ying Chen has joined the channel

Ying Chen (01:40:07): > Hi. I have an error in build on palomino4 that is related to _cache/, could anyone help here? > > --- Start generating read class files --- > [E::hts_hopen] Failed to open file F:\biocbuild\ExperimentHub_cache/29846dc07189_3844 > [E::hts_open_format] Failed to open file "F:\biocbuild\ExperimentHub_cache/29846dc07189_3844" : Exec format error > Quitting from lines 153-155 (bambu.Rmd) > Error: processing vignette 'bambu.Rmd' failed with diagnostics: > BiocParallel errors > 1 remote errors, element index: 1 > 0 unevaluated and other errors > first remote error: > Error in value[[3L]](cond): failed to open BamFile: failed to open SAM/BAM file > file: 'F:\biocbuild\ExperimentHub_cache/29846dc07189_3844' > > Thank you!

Lori Shepherd (07:38:16) (in thread): > Thank you for making us aware – we also saw your post on the bioc-devel mailing list. Once we are finished with some post release tasks we can look into this for you. Please be patient as this is a busy time for core team.

Aidan Lakshman (16:42:48) (in thread): > Update: bug is fixed (will be pushed to release soon). Refactored some code and found out that there was an accidental<=that should have been an<while allocating memory for a 2D array–little tough to explain the exact source but essentially an extra row was accidentally allocated and then never freed, causing problems later on. Good news is everything is building successfully, and we now have github actions and R-Hub set up for build testing. Thanks again!

Ying Chen (20:48:42) (in thread): > Thank you!!

2022-11-10

Alan Murphy (15:11:52): > Hey! I’m the maintainer ofMungeSumstatsand have noticed some fails on nightly builds. Firstly, it was passing all checks on release (3.16) but has now failed on Iconway with the following: > > Quitting from lines 251-261 (MungeSumstats.Rmd) > Error: processing vignette 'MungeSumstats.Rmd' failed with diagnostics: > Install 'SNPlocs.Hsapiens.dbSNP155.GRCh37' to use 'GRCh37' as 'ref_genome' and 155 as dbSNP version > > Not sure if SNPlocs.Hsapiens.dbSNP155.GRCh37 needs to be installed manually on your end or if this will resolve itself? > > Secondly, it is failing on dev (3.17), all machines since the release down to a Latext error which I can’t seem to replicate: > > * checking PDF version of manual ... WARNING > LaTeX errors when creating PDF version. > This typically indicates Rd problems. > LaTeX errors found: > ! Paragraph ended before \Rd@code@wrk was complete. > <to be read again> > \par > l.4301 > > ! Paragraph ended before \Rd@code@wrk was complete. > <to be read again> > \par > l.5356 > > * checking PDF version of manual without index ... ERROR > * DONE > > Oddly, this is the same version of MSS that’s passing on release (except on Iconway right now)

Andres Wokaty (15:24:56) (in thread): > I’ll have a look at these. The latex error is happening to other packages.

Alan Murphy (15:25:58) (in thread): > Thank you very much!

2022-11-13

Ludwig Geistlinger (18:09:01): > One question about the recent transition of my BioPlex package from software package to experiment data package. It looks like the package has been released as adata packagewith Bioc3.16 as intended, but it seems that aconcurrent version of the package listed as a software packagestill exists in 3.16?

Lori Shepherd (18:16:37): > We can look into it. Thanks for making us aware.

Ludwig Geistlinger (18:17:03): > Thanks!

2022-11-14

Henrik Bengtsson (11:35:00) (in thread): > Thanks for doing this (i.e. dropping--no-vignettesfrom the Linux checks). Next step would be to check with*R_CHECK_DEPENDS_ONLY_VIGNETTES*=true. I’ve createdhttps://github.com/Bioconductor/BBS/issues/242for this and to move the discussion there - Attachment: #242 WISH: Check for forgotten package dependencies in package vignettes > For Bioconductor devel (3.17), please consider running checks with: > > > *R_CHECK_DEPENDS_ONLY_VIGNETTES*=true > > > This will make R CMD check check package vignettes in a sandboxed environment where the package library only holds packages that are declared as dependencies (and the dependencies of the dependencies, etc.). > > PS. This requires a recent R-devel version, since there was a bug related to *R_CHECK_DEPENDS_ONLY_VIGNETTES* preventing this to work for vignettes. > > Example > > For example, compare all OK results in https://bioconductor.org/checkResults/devel/bioc-LATEST/MOSim/nebbiolo1-checksrc.html with: > > > ## Just to show that I don't have a ~/.R/check.Renviron in play > $ export R_CHECK_ENVIRON=non-existing > > $*R_CHECK_DEPENDS_ONLY_VIGNETTES*=true R --vanilla CMD check MOSim_1.13.0.tar.gz > * using log directory ‘/tmp/bioc/MOSim.Rcheck’ > * using R Under development (unstable) (2022-11-13 r83341) > * using platform: x86_64-pc-linux-gnu (64-bit) > * using session charset: UTF-8 > * checking for file ‘MOSim/DESCRIPTION’ ... OK > * this is package ‘MOSim’ version ‘1.13.0’ > * package encoding: UTF-8 > * checking package namespace information ... OK > * checking package dependencies ... > Warning: unable to access index for repository [https://CRAN.R-project.org/src/contrib](https://CRAN.R-project.org/src/contrib): > cannot open URL '[https://CRAN.R-project.org/src/contrib/PACKAGES](https://CRAN.R-project.org/src/contrib/PACKAGES)' > > OK > * checking if this is a source package ... OK > * checking if there is a namespace ... OK > * checking for executable files ... OK > * checking for hidden files and directories ... OK > * checking for portable file names ... OK > * checking for sufficient/correct file permissions ... OK > * checking whether package ‘MOSim’ can be installed ... OK > * checking installed package size ... NOTE > installed size is 5.1Mb > sub-directories of 1Mb or more: > data 4.2Mb > * checking package directory ... OK > * checking ‘build’ directory ... OK > * checking DESCRIPTION meta-information ... OK > * checking top-level files ... OK > * checking for left-over files ... OK > * checking index information ... OK > * checking package subdirectories ... OK > * checking R files for non-ASCII characters ... OK > * checking R files for syntax errors ... OK > * checking whether the package can be loaded ... OK > * checking whether the package can be loaded with stated dependencies ... OK > * checking whether the package can be unloaded cleanly ... OK > * checking whether the namespace can be loaded with stated dependencies ... OK > * checking whether the namespace can be unloaded cleanly ... OK > * checking loading without being on the library search path ... OK > * checking dependencies in R code ... OK > * checking S3 generic/method consistency ... OK > * checking replacement functions ... OK > * checking foreign function calls ... OK > * checking R code for possible problems ... OK > * checking Rd files ... OK > * checking Rd metadata ... OK > * checking Rd cross-references ... OK > * checking for missing documentation entries ... OK > * checking for code/documentation mismatches ... OK > * checking Rd \usage sections ... OK > * checking Rd contents ... OK > * checking for unstated dependencies in examples ... OK > * checking contents of ‘data’ directory ... OK > * checking data for non-ASCII characters ... OK > * checking data for ASCII and uncompressed saves ... OK > * checking sizes of PDF files under ‘inst/doc’ ... OK > * checking installed files from ‘inst/doc’ ... OK > * checking files in ‘vignettes’ ... OK > * checking examples ... > OK > * checking for unstated dependencies in ‘tests’ ... OK > * checking tests ... > Running ‘testthat.R’ > OK > * checking for unstated dependencies in vignettes ... OK > * checking package vignettes in ‘inst/doc’ ... OK > * checking running R code from vignettes ... > ‘MOSim.Rnw’ using ‘UTF-8’... OK > NONE > * checking re-building of vignette outputs ... ERROR > Error(s) in re-building vignettes: > ... > --- re-building ‘MOSim.Rnw’ using knitr > Quitting from lines 8-10 (MOSim.Rnw) > Error: processing vignette 'MOSim.Rnw' failed with diagnostics: > there is no package called 'BiocStyle' > --- failed re-building ‘MOSim.Rnw’ > > SUMMARY: processing the following file failed: > ‘MOSim.Rnw’ > > Error: Vignette re-building failed. > Execution halted > > * checking PDF version of manual ... OK > * checking for non-standard things in the check directory ... OK > * checking for detritus in the temp directory ... OK > * DONE > > Status: 1 ERROR, 1 NOTE > > > Expected outcome > > From doing many revdep checks, I’d say the most common mistakes are to forget Suggests: BiocStyle and Suggests: codetools, but there are of course many others too.

Henrik Bengtsson (11:51:14) (in thread): > BTW, here’s an example of a bug thatR CMD checkon the Linux machine now picks up from running the vignettes:http://bioconductor.org/checkResults/release/bioc-LATEST/CopywriteR/nebbiolo2-checksrc.html

Lluís Revilla (12:04:51) (in thread): > There have been some changes in r-devel today that might help detecting bugs in vignettes too:https://developer.r-project.org/blosxom.cgi/R-devel/NEWS/2022/11/14

Henrik Bengtsson (14:43:22) (in thread): > Yes, those NEWS entries are related to this; it’s Sebastian Meyer (R Core) has been working on improving things. As a follow up on this original thread, I emailed Kurt Hornik (R Core, CRAN, and CRAN checks) about*R_CHECK_DEPENDS_ONLY_VIGNETTES*not working as expected (like*R_CHECK_DEPENDS_ONLY_TESTS*,*R_CHECK_DEPENDS_ONLY_EXAMPLES*, and*R_CHECK_DEPENDS_ONLY_DATA*). He cc:ed in Sebastian, who independently had discovered related problems when investigate whyknitr-based vignettes were not fully tested (https://github.com/yihui/knitr/pull/2036). Sebastian then did a push and fixed several issues.

2022-11-16

Alan Murphy (05:23:26) (in thread): > @Andres Wokatyjust to update you, the release version of MSS (1.6.0) is now passing checks however, the dev version is still failing with the same issue.

Lluís Revilla (06:20:30): > Yesterday I was helping a macOS user trying to install genefilter (well, really DESeq2), but BiocManager reported something along > > Warning: unable to access index for repository[https://bioconductor.org/packages/3.16/bioc/bin/macosx/big-sur-arm64/contrib/4.2](https://bioconductor.org/packages/3.16/bioc/bin/macosx/big-sur-arm64/contrib/4.2): > cannot open URL '[https://bioconductor.org/packages/3.16/bioc/bin/macosx/big-sur-arm64/contrib/4.2/PACKAGES](https://bioconductor.org/packages/3.16/bioc/bin/macosx/big-sur-arm64/contrib/4.2/PACKAGES)' > Warning: unable to access index for repository > Package which is only available in source form > > and couldn’t install genefilter due to gfortran. The R version was 4.2.1 and even after installing gfortran frommac.r-project.org/toolsit didn’t work . I know that producing binaries for macOS have some delays/problems, is there any workaround or the user should just wait?

Martin Morgan (06:26:48) (in thread): > I clicked just now on the ‘PACKAGES’ url in the error message and it opened in my browser. Also > > > url = "[https://bioconductor.org/packages/3.16/bioc/bin/macosx/big-sur-arm64/contrib/4.2/PACKAGES](https://bioconductor.org/packages/3.16/bioc/bin/macosx/big-sur-arm64/contrib/4.2/PACKAGES)" > > dcf = read.dcf(url(url)) > > worked; are there network issues that are blocking access, either in the browser or from R?

Lluís Revilla (06:29:45) (in thread): > It was a classroom of 20 people and several macOS and only this user had this problem. We tried with several hours in between and the error persisted so I don’t think it was a connectivity problem.

Vince Carey (08:25:34) (in thread): > @Alex Mahmoudcan you have a look at this?@Lluís Revilla– We are trying to distribute resources in a way that is efficient for remote sites and we do not have an international testing scheme to check that expected connections are alive/suitably configured/have appropriate fallbacks. I do not know whether manual use of a remote mirror could give better results;@Lori Shepherdwould mac binaries be on the mirrors?

Lori Shepherd (08:26:58) (in thread): > Depends on how a mirror maintainer set up the sync but generally yes they should be

Vince Carey (08:28:51) (in thread): > @Lluís Revillayou say “only this user” – does it mean only the M1 mac user had the problem?

Vince Carey (08:29:08) (in thread): > The other intel macs were ok?

Lluís Revilla (08:31:29) (in thread): > Thanks@Vince Carey. I initially thought that the binaries were not built yet for this package/OS. I don’t know, if this mac was a M1 or not. I didn’t thought of checking this, but as far as I know other macs were ok.

Vince Carey (08:39:03) (in thread): > the big-sur-arm64 in the path indicates the request is coming from an M1 mac. we do have 3.16 binaries for that platform but the distribution process is quite new and there might be gaps.

Lluís Revilla (08:46:12) (in thread): > in this case, I’ll try to collect more information in the next training session with a different cohort of people next week. But as we also have a RStudio cloud we might not see it

Kasper D. Hansen (09:16:08) (in thread): > Luis, so for Bioc 3.15 / R 4.2 (previous version) we did not have binaries. For Bioc 3.16 / R 4.2 we now have binaries, so versioning is critical here. However, the URL contains3.16suggesting it is trying to get the right Bioc version. Like@Martin Morgan, I can resolve the URL at home on my mac.

Lluís Revilla (09:25:16) (in thread): > I am actually not sure if Bioconductor version was 3.16 or 3.15, but if it was 3.15 it wouldn’t resolve to binaries and BiocManager would install from source? In that case the user wouldn’t be able to install genefilter without having gfortran, which was what I tried to solve. Thanks all!

Kasper D. Hansen (22:51:52) (in thread): > If it was Bioc 3.15, BiocManager would (correctly) try to install genefilter from source.

2022-11-18

Ludwig Geistlinger (15:13:48) (in thread): > Hi@Lori Shepherdjust wanted to shortly check back on this. Sorry for pushing a little bit on this one, but we have a paper that we are currently revising for publication and it would be great if this could be resolved prior to resubmission. In particular, we are providing the package landing page ashttps://bioconductor.org/packages/BioPlexin the availability section of the paper and it would be great if this could point to the data package in release as opposed to pointing to the artifact that seemed to have remained from the software package. Many thanks!

Lori Shepherd (17:41:30) (in thread): > We thought we had – its the hard part about switching – there are a lot of places that need to be cleared of caches in order to get it removed – It looks like it no longer appears in the software VIEWS so I’ll try removing the pages again —

Ludwig Geistlinger (17:46:59) (in thread): > Thank you, Lori!:pray:

2022-11-20

Mikhail Dozmorov (10:45:17): > I’m trying to update vignette in an AnnotationHub data package, excluderanges. Any attempts to bump its current 0.99.6 version and push to the “RELEASE_3_16” branch fail with “Error: Illegal version bump”. I can push without version bump, but these changes are not propagated. These changes do propagate in the “devel” branch with automatic bump to version 0.99.7. I’ve been monitoring build reportshttps://bioconductor.org/checkResults/, reading about versionshttps://contributions.bioconductor.org/versionnum.html, interacting with Bioc GitHub, nothing works. Same issue with CTCF data package. This has been going for weeks. It must be very simple, I ran out of ideas, any suggestions?

Marcel Ramos Pérez (10:50:06): > AFAIU, annotation data packages do not work the same way as software packages. Bumps and propagation have to be done manually. We are working to make them behave the same way but for now a team member can work with you to get those updated.

Mikhail Dozmorov (10:57:25) (in thread): > Thanks, Marcel. Will wait for help and hope it’ll be easier in the future.

Lori Shepherd (18:05:25) (in thread): > should be all set now

Ludwig Geistlinger (18:15:47) (in thread): > Great, thanks! Only thing that I see that the software packages would still show up when entering “Bioplex” into the search bar, but would then lead to a “page not found” when clicking on the corresponding links

2022-11-21

Luke Zappia (02:10:55) (in thread): > If your package was accepted in the last release it should have a version starting with 1 (probably1.0.0) for theRELEASE_3_16branch and1.1.0for the currentDEVELbranch. You probably need to do agit pullfrom Bioconductor to get the correct version numbers.

Mikhail Dozmorov (07:18:39) (in thread): > Apparently, versioning for data packages is different. I tried 1.0.0 and combinations of 2nd and 3rd numbers. Pulling from Bioc also gives the old version numbers. As Marcel mentioned, updating data packages requires manual intervention. At least I can update Github/pkgdown. Thanks, Luke!

Luke Zappia (07:19:56) (in thread): > Ah, sorry I missed that it was a data package. Hope you get it sorted.

Lori Shepherd (07:43:11) (in thread): > experiment data packages work the same way as software, its just annotation packages that are different. We don’t automatically bump and branch annotations as it implies annotations changed when often this isn’t the case.

Lori Shepherd (07:43:23) (in thread): > replied via email – you should be all set to push changes now

Marcel Ramos Pérez (10:12:36) (in thread): > Thank you Lori

Lori Shepherd (10:14:58) (in thread): > Revamping annotation hooks is first priority after I get back from Thanksgiving so hopefully easier for all very soon

2022-11-22

Andres Wokaty (14:18:55) (in thread): > Hi Alan, I’m still looking into this error; however, I noticed that the vignettes are installing packages, which they should not be doing (seehttps://contributions.bioconductor.org/docs.html#installation). - Attachment (contributions.bioconductor.org): Chapter 12 Documentation | Bioconductor Packages: Development, Maintenance, and Peer Review > Package documentation is important for users to understand how to work with your code. 12.1 Bioconductor documentation minimal requirements: a vignette in Rmd or Rnw format with executable code…

2022-11-23

Ying Chen (02:05:51) (in thread): > Hi@Lori Shepherd, have you had any chance to look into this issue? Thank you!

Alan Murphy (09:48:37) (in thread): > Hi Jennifer, thanks for continuing to look into this for me. However, there isn’t anywhere in the vignettes where packages are installed that I can see? Line 213-216 and 222-225 inMungeSumstats.Rmdaren’t actually run - they are just formatted as code. I just have them to explain to users since these aren’t dependencies of the package but are needed to run certain functions.

Lori Shepherd (13:44:15) (in thread): > I started to but didn’t get time to finish before the holiday. I’ve scheduled time to look at it again on Monday.

2022-11-28

Jianhai Zhang (02:55:02): > I have build errors (http://bioconductor.org/checkResults/devel/bioc-LATEST/spatialHeatmap/nebbiolo1-buildsrc.html) and have tried the docker (https://www.bioconductor.org/help/docker/) to reproduce them. Can any adminstrator install ‘gs’ (https://superuser.com/questions/997002/installing-ghostscript-in-linux) on the Bioc devel? Since I got the following error on the docker: > > sh: 1: gs: not found > Error in PostScriptTrace(p1, p2) : sorry, ‘gs’ cannot be found

Vince Carey (06:30:22): > within container (e.g.,docker run -ti bioconductor/bioconductor_docker:devel bashand you have a shell) > > apt-get update > apt-get install ghostscript > which ghostscript >

Vince Carey (06:36:04): > If you used docker as given above, after you install ghostscript in the > container, exit the container and use your shell to interrogate your > recent creations: > > $ docker ps -a |head > CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES > 9b395682355c bioconductor/bioconductor_docker:devel "bash" 3 minutes ago Exited (0) About a minute ago peaceful_carver > > That container id can be used in a commit, to produce a new image with a name > of your choice. I used bioc_dev:with_gs > > $ docker commit 9b395682355c bioc_dev:with_gs > sha256:c2b81b7b069301b1f2529f1df3339ff17910ffd36ba2417cf0750c15cc335cc1 > > I now see that in my list of images > > $ docker images |head -4 > REPOSITORY TAG IMAGE ID CREATED SIZE > bioc_dev with_gs c2b81b7b0693 7 seconds ago 4.26GB > bioconductor/bioconductor_docker devel 8545645f4864 2 days ago 4.15GB > bioconductor/bioconductor_docker <none> 930ae8a5455e 2 weeks ago 4.16GB > > I can now use the ghostscript-enabled container: > > $ docker run -ti bioc_dev:with_gs bash > root@63e88f5a6f4f:/# which ghostscript > /usr/bin/ghostscript >

Mikhail Dozmorov (21:09:21): > Anothergfortranissuebreaking things. It converges toDNAcopyand packages depending on it (QDNAseq,Rsamtools). This started after upgrading R to 4.2.2 and Bioconductor to 3.16. On an Intel Mac, latest macOS Ventura. Herehttps://github.com/nullranges/nullranges/issues/20I describe all I could find for a workaround using bioconductor:devel Docker image. Can it be solved in release? - Attachment: #26 Issue installing HiCcompare - Attachment: #20 gfortran required for DNAcopy

2022-11-29

Lori Shepherd (08:58:40) (in thread): > its odd that this appears on one windows machine and not the other. I traced it down to the BamFile not being able to be read – I’m not sure if its a BiocParallel issue or a file issue. I have reset the cache on the system and am hoping on tonights build that it will clear up with a fresh download of the resources. if not I can look at it again on wed.

Vince Carey (10:07:37): > Is this really an issue on intel macs? On the M1 mac you need to introduce some dylib manipulations to produce a proper binary and this is done in release as you can see athttps://bioconductor.org/checkResults/3.16/bioc-mac-arm64-LATEST/DNAcopy/kjohnson-buildbin.html– go to the bottom of that page.

Vince Carey (10:09:52): > maybe the same thing is needed in ventura?

Ying Chen (20:43:35) (in thread): > Thank you for looking into this!@Lori ShepherdDo let me know if there are anything I can help with!

2022-11-30

Jianhai Zhang (04:22:58) (in thread): > Thank you.

Jianhai Zhang (04:26:07): > Can any Bioc administrator install theghostscripton the devel branch?

Aidan Lakshman (07:32:23) (in thread): > Hey Jianhai, I had issues with this as well during development of our package—if you’re developing using GitHub I’ve set up a GitHub action file that automatically tests build on windows/Linux/Mac without failing at normal latex dependencies like ghostscript:https://github.com/ahl27/SynExtend/blob/master/.github/workflows/rpkgchecks.ymlI removed the OSX package check, but you should be able to add it in below line 19. > > Hopefully that helps with your testing pipeline. Sorry for the slightly off topic response, but based on your other comments I thought it may be helpful to you.

Kasper D. Hansen (10:09:39): > May I ask why? Just curious.

Jianhai Zhang (11:32:19) (in thread): > It is a dependency of my package.

Kasper D. Hansen (12:12:06) (in thread): > But why is it a dependency?

Jianhai Zhang (13:05:10) (in thread): > I tested my package on bioc docker and got the error: > > sh: 1: gs: not found > Error in PostScriptTrace(p1, p2) : sorry, ‘gs’ cannot be found

Lori Shepherd (13:24:38) (in thread): > It looks like the caching error has cleared up. There is now a different ERROR in the check portion of the package that is consistent in release and devel. Sorry for the delay in fixing the caching error. cheers

Jianhai Zhang (13:46:50) (in thread): > Hi Aidan, thanks for your reply. It is useful to me, not off-topic.

Andres Wokaty (14:26:10) (in thread): > We fixed the gfortran path issues with our Mac ARM64 (M1) builder, so the new binary forDNAcopythat will be available Sunday will work for Mac ARM64 machines. > > The issue you’re experiencing is on an Intel Mac with Ventura withgfortran?

Hervé Pagès (14:30:40): > What does it mean to install ghostscript on the devel branch? If you mean on the devel build machines, it’s already there and the reason your package fails on these machines doesn’t seem to have anything to do with ghostscript:https://bioconductor.org/checkResults/3.17/bioc-LATEST/spatialHeatmap/If you mean on the bioc docker image then you should ask on the#containerschannel. But didn’t@Vince Careyexplain how to install ghostscript on your local image already? If things were not clear or didn’t work as expected, then you should ask for clarifications. But please provide some context/details when you do so. Also show what you’ve done, what didn’t work, what error you got, etc… Thx

Ying Chen (22:20:08) (in thread): > Thank you so much@Lori Shepherdfor your patience and continuous support!

2022-12-01

Alan Murphy (11:53:41) (in thread): > Hey! Just as a bit of an update, I made some changes to the package (v1.7.7) which solved some issues I was getting locally but these changes done some to be reflected in the latest build report despite the version of the package being correct. The change has to do with a temp directory, could the session be restarted so this is reset?

Andres Wokaty (11:54:51) (in thread): > I noticed you you were making some changes. Should I remove the temp directory?

Alan Murphy (11:56:04) (in thread): > Please! Hopefully this will fix the issues we were noticing but let’s see!

2022-12-04

Jianhai Zhang (00:14:28) (in thread): > I am in the process of updating the package and on the Windows Server 2022 Datacenter it says “there is no package called ‘fastcluster’”, so could you install it?

Hervé Pagès (00:51:46) (in thread): > Seems likespatialHeatmapdepends indirectly onfastcluster(i.e. via another package). The build system normally takes care of installing direct and indirect dependencies automatically. However I know that Jen updated R-devel recently on the devel builders and sometimes it takes a couple of build cycles for things to come back to normal on Windows after such update. > Looking at the build logs I see that the build system failed to reinstallfastclusteron palomino3. The logs contain: > > trying URL '[https://cran.rstudio.com/bin/windows/contrib/4.3/fastcluster_1.2.3.zip](https://cran.rstudio.com/bin/windows/contrib/4.3/fastcluster_1.2.3.zip)' > Content type 'application/zip' length 233931 bytes (228 KB) > ================================================== > downloaded 228 KB > > package 'fastcluster' successfully unpacked and MD5 sums checked > Error in unpackPkgZip(foundpkgs[okp, 2L], foundpkgs[okp, 1L], lib, libs_only, : > ERROR: failed to lock directory 'F:\biocbuild\bbs-3.17-bioc\R\library' for modifying > Try removing 'F:\biocbuild\bbs-3.17-bioc\R\library/00LOCK' > Calls: installNonTargetPkg ... install.packages -> .install.winbinary -> unpackPkgZip > Execution halted > > @Andres WokatyWhen you get a chance, can you go on palomino3 and remove that00LOCKfile? Sometimes it just goes away after a couple of build cycles but not always. Thanks!

Alan Murphy (06:30:04) (in thread): > Hey Jennifer, that same error still appears to be occuring, if you go on to the machine and run R CMD Check manually does it still occur? I can’t really understand why my last fix didn’t resolve it especially since you remove the temp directory

2022-12-05

Atuhurira Kirabo Kakopo (08:51:02): > @Atuhurira Kirabo Kakopo has joined the channel

Andres Wokaty (15:12:05) (in thread): > I still get the same error when I runR CMD checkmanually on nebbiolo1. The temp directory was just in/tmpand no where else?

2022-12-06

Mike Smith (10:45:50): > Which platform builds the package source tarballs that are distributed from the landing pages? Is it Windows? I was just looking atlpsymphonyand my output from R CMD build is different from what’s available from the landing page. I was wondering if it might be acleanupvscleanup.winsituation.

Lori Shepherd (10:49:48): > I’m not sure what difference you see as it seems consistent with the version numbers in the build report that are on the current landing page?

Mike Smith (11:16:05): > Turns out, if you runR CMD buildwith the--no-build-vignettesargument it might do more than just not build the vignettes. This screen shot is of two tarballs, only difference is if I provide that argument. No only are the vignettes missing on the left, but thesrcfolder is 100MB smaller because building the package adds a tonne of other stuff. - File (PNG): image.png

Mike Smith (11:18:20): > This is in the context of this ridiculously long standing bug (https://support.bioconductor.org/p/106100/#106268) that has arrisen on our cluster again. I wanted to understand why some weird files are being left in the package tarball, and was confused when I couldn’t reproduce it myself. However, that’s because I was providing--no-build-vignettes, which you don’t do on the build system.

Lori Shepherd (11:27:10): > interesting. yes we build the vignette in the R CMD build stage for completeness and then don’t rebuild it during R CMD check (at least I think this is still the case)

Alan Murphy (12:38:37) (in thread): > It should be? The only saved directory is fromtempdir(). It’s still very unclear to me why these errors are occurring, For example on my local MAC I can run: > > (base) alanmurphy@Alans-MacBook-Pro-2 R_packages % R CMD build --keep-empty-dirs --no-resave-data MungeSumstats > * checking for file 'MungeSumstats/DESCRIPTION' ... OK > * preparing 'MungeSumstats': > * checking DESCRIPTION meta-information ... OK > * installing the package to build vignettes > * creating vignettes ... OK > * checking for LF line-endings in source and make files and shell scripts > * checking for empty or unneeded directories > WARNING: directory 'MungeSumstats/tests/testthat/_snaps' is empty > * looking to see if a 'data/datalist' file should be added > * building 'MungeSumstats_1.7.8.tar.gz' > > without issue and yet this fails on merida1

Hervé Pagès (13:27:01): > @Mike SmithTo answer your original questions we propagate the source tarballs obtained on the Linux builders. The exact command we use to produce these tarballs is: > > /home/biocbuild/bbs-3.17-bioc/R/bin/R CMD build --keep-empty-dirs --no-resave-data <pkgdir> > > See for example:https://bioconductor.org/checkResults/3.17/bioc-LATEST/a4Base/nebbiolo1-buildsrc.html

Vince Carey (15:17:38): > I am trying to find time to produce a diagram that addresses the “life cycle” of a bioconductor package in relation to build processes, dependencies within and outside of bioc, etc. starts with a git repo (source, with various branches), splits into release/devel on platforms, …

2022-12-14

Alan Murphy (12:25:46) (in thread): > Hey@Andres Wokaty, I’m still working to try and fix the errors but I notice a required package isn’t installed on nebbiolo (SNPlocs.Hsapiens.dbSNP155.GRCh37):https://bioconductor.org/checkResults/3.17/bioc-LATEST/MungeSumstats/nebbiolo1-buildsrc.htmlWorth mentioning MSS also needs all of the following to run checks: > > SNPlocs.Hsapiens.dbSNP144.GRCh37, > SNPlocs.Hsapiens.dbSNP144.GRCh38, > SNPlocs.Hsapiens.dbSNP155.GRCh37, > SNPlocs.Hsapiens.dbSNP155.GRCh38, > BSgenome.Hsapiens.1000genomes.hs37d5, > BSgenome.Hsapiens.NCBI.GRCh38,

Andres Wokaty (16:29:06) (in thread): > Thanks for noting this. I’m confused as to why these aren’t being automatically installed, so I am discussing with other members of the core team before I manually install.

Alan Murphy (17:16:31) (in thread): > Thank you!!

2022-12-15

Andres Wokaty (15:11:06) (in thread): > I updated R on the devel builders on Monday, which is likely why it needed another day to resolve. (We do this periodically throughout the development cycle, so you might see it again the next update.) It seems that it’s now passing on Linux, but there are some vignette-related issues on Windows and Mac, fortunately not the original Latex errors, though.

Alan Murphy (15:32:31) (in thread): > That’s good news on Linux at least! The other two appear to be the same issue I think I managed to solve in previous push. Could you possibly try removing the temp directory from these two again? I believe the issue might be cached files which have since been updated. If that doesn’t work I’ll try and replicate the issues on my end again

Andres Wokaty (15:59:51) (in thread): > I missed the window for the start of the builds today, so I’m going to remove them tomorrow before the builds start.

Alan Murphy (16:00:14) (in thread): > Thank you!!

2022-12-20

Alan Murphy (11:54:59) (in thread): > Hey@Andres WokatyI have made a fix in the dev branch which I believe will sort at least the linux error so hopefully that will be apparent tomorrow. However, in a slightly unrelated issue - I see MSS is now failing on linux in the release branch even though no changes have been pushed. It seems to be killed on a unit test when loading the dbSNP reference genome: > > * checking tests ... > Running 'testthat.R' > ERROR > Running the tests in 'tests/testthat.R' failed. > Last 13 lines of output: > - 101 unique variants > - 0 genome-wide significant variants (P<5e-8) > - 1 chromosomes > Checking for multi-GWAS. > Checking for multiple RSIDs on one row. > Checking SNP RSIDs. > Checking for merged allele column. > Checking A1 is uppercase > Checking A2 is uppercase > Checking for correct direction of A1 (reference) and A2 (alternative allele). > Loading SNPlocs data. > Loading reference genome data. > Preprocessing RSIDs. > Validating RSIDs of 101 SNPs using BSgenome::snpsById... > Killed > > More here:https://bioconductor.org/checkResults/3.16/bioc-LATEST/MungeSumstats/nebbiolo2-checksrc.html. Is there any reason this could be appearing only now?

2023-01-12

Martin Grigorov (13:05:08): > @Martin Grigorov has joined the channel

2023-01-13

Alan Murphy (10:22:17): > Hey! I’m the maintainer ofMungeSumstats(MSS) which has been failing on the dev bioconductor for the last couple of weeks. It is failing to build on MAC and Windows. Despite best efforts I can’t replicate the errors locally - myself and a colleague has tried to do so on Mac which hasn’t resulted in any issues (on R 4.2 with the dev version of MSS): > > R CMD build MungeSumstats > * checking for file 'MungeSumstats/DESCRIPTION' ... OK > * preparing 'MungeSumstats': > * checking DESCRIPTION meta-information ... OK > * installing the package to build vignettes > * creating vignettes ... OK > * checking for LF line-endings in source and make files and shell scripts > * checking for empty or unneeded directories > * looking to see if a 'data/datalist' file should be added > * building 'MungeSumstats_1.7.12.tar.gz' > > I think the issue is due to changes I made to use the ensembl chain file by default rather than ucsc due to licensing issues. The function ishereand should try to download the latest chain file from ensembl (we do this so we know it’s up to date) but if this fails or there is no internet connection, a local version of the chain file in MSS should be used. I have tested running both on and off-line on my Mac to check this isn’t the issue either. The output of R CMD Build from the Bioconductor Mac machine is (full reporthere): > > checking for file 'MungeSumstats/DESCRIPTION' ... OK > * preparing 'MungeSumstats': > * checking DESCRIPTION meta-information ... OK > * installing the package to build vignettes > * creating vignettes ... ERROR > --- re-building 'MungeSumstats.Rmd' using rmarkdown > trying URL '[ftp://ftp.ensembl.org/pub/assembly_mapping/homo_sapiens/GRCh37_to_GRCh38.chain.gz](ftp://ftp.ensembl.org/pub/assembly_mapping/homo_sapiens/GRCh37_to_GRCh38.chain.gz)' > Content type 'unknown' length 285250 bytes (278 KB) > ================================================== > Quitting from lines 618-623 (MungeSumstats.Rmd) > Error: processing vignette 'MungeSumstats.Rmd' failed with diagnostics: > expected 11 elements in header, got 1, on line 4 > --- failed re-building 'MungeSumstats.Rmd' > > --- re-building 'OpenGWAS.Rmd' using rmarkdown > --- finished re-building 'OpenGWAS.Rmd' > > --- re-building 'docker.Rmd' using rmarkdown > [WARNING] Could not parse YAML metadata at line 19 column 1: :8:28: Unexpected ' > ' > [WARNING] This document format requires a nonempty <title> element. > Please specify either 'title' or 'pagetitle' in the metadata, > e.g. by using --metadata pagetitle="..." on the command line. > Falling back to 'docker.knit' > --- finished re-building 'docker.Rmd' > > SUMMARY: processing the following file failed: > 'MungeSumstats.Rmd' > > Error: Vignette re-building failed. > Execution halted > > So it seems to have issues using the ensembl file once downloaded but since I can’t replicate the issue I can’t seem to debug it.@Andres Wokatywas helping me work through this at the end of last year and I think we sorted some issues (it wasn’t passing on linux before but now is) but I feel a little stuck now. If anyone has had a similar error, it would be great to hear how you got around it?

Vince Carey (19:23:39): > My suggestion here would be to avoid using ensembl on the web in any test-vulnerable component of any package. If u must do this ensure you are fault tolerant and have longer timeout limit than default. Prefer realistic in-package or in-cache mocks for query results

2023-01-16

Alan Murphy (07:43:23): > Hey Vince! So I do have a version of ensembl in the package which should be triggered to be used if there are any issues downloading the online version and if you look at the printed messages it looks like it downloads fine but the issue is actually with reading the file -https://bioconductor.org/checkResults/3.17/bioc-LATEST/MungeSumstats/merida1-buildsrc.html: > > * creating vignettes ... ERROR > --- re-building 'MungeSumstats.Rmd' using rmarkdown > trying URL '[ftp://ftp.ensembl.org/pub/assembly_mapping/homo_sapiens/GRCh37_to_GRCh38.chain.gz](ftp://ftp.ensembl.org/pub/assembly_mapping/homo_sapiens/GRCh37_to_GRCh38.chain.gz)' > Content type 'unknown' length 285250 bytes (278 KB) > ================================================== > > It’s very hard to know what the issue is though since I can’t replicate it

Jonathan Griffiths (08:49:39) (in thread): > The error is about reading the file by the looks of it: > > failed with diagnostics: > expected 11 elements in header, got 1, on line 4 > > I wonder if there is an issue with thesedcommand that I put in there?

Alan Murphy (09:24:40) (in thread): > I think there was originally - you were using sed -i but I changed it to sed -E which shouldn’t cause an issue across OS - I can’t find the link now where they discuss the issues with sed -i

Alan Murphy (09:25:02) (in thread): > Annoyingly though this works locally on my mac but not on Bioconductors…

Mike Smith (11:07:46) (in thread): > I haven’t spent much time looking at this, but maybe it’s significant that the contents of the chain file is printed to output on Windows (https://bioconductor.org/checkResults/3.17/bioc-LATEST/MungeSumstats/palomino3-buildsrc.html). That feels like something isn’t working properly on multiple platforms to me. > > Perhaps you don’t need your call tosedat all. It looks like the original issue has been fixed in rtracklayer (https://github.com/lawremi/rtracklayer/issues/23) and pushed to Bioconductor (https://code.bioconductor.org/browse/rtracklayer/commit/894b6699a61a1149071fbbd6037d9160aed8b02b)

2023-01-17

Martin Grigorov (06:51:02): > Hello! What ismeat(in~/bbs-VER-bioc/meat/) folder athttps://github.com/Bioconductor/BBS/blob/master/Doc/Prepare-Ubuntu-20.04-HOWTO.md#testing-2? It is not introduced earlier and it is unclear to me how to create it with the needed sub-folders.

Lori Shepherd (07:31:17): > @Andres Wokatymight have a better description but I believe it contains a checkout of packages and the output of running all the build commands for install, build, check, etc ….

Alan Murphy (08:53:41) (in thread): > Hey Mike, that appears to have resolved my work around locally, I’ll push it and see if it sorts out on Bioconductor’s end. Thanks for the suggestion!

Andres Wokaty (11:04:35) (in thread): > This is correct.

2023-01-18

Hervé Pagès (19:34:15) (in thread): > It will be automatically created and populated the first time you run the builds on your machine. Note that it’s only mentioned in section 3 of Prepare-Ubuntu-20.04-HOWTO.md. This section documents additional things to do after a first succesful run so themeat/folder is assumed to exist at that point. But we are not there yet so you can ignore that section for now.

Hervé Pagès (19:46:17) (in thread): > Now clarified at the end of section 2:https://github.com/Bioconductor/BBS/blob/master/Doc/Prepare-Ubuntu-20.04-HOWTO.md#26-first-build-report

2023-01-19

Martin Grigorov (01:51:23) (in thread): > Thank you,@Hervé Pagès!

2023-01-20

Yikun Jiang (03:13:39): > @Yikun Jiang has joined the channel

Vince Carey (16:17:07) (in thread): > is this sorted?

2023-01-23

Alan Murphy (05:41:07) (in thread): > Hey! Yep sorted now, just waiting for a small fix to propagate today for mac to pass checks but windows and linux are now passing (v1.7.16)

Alan Murphy (11:32:39) (in thread): > Passing on all platforms now, thank you everyone!

2023-01-24

Martin Grigorov (03:26:53): > Hello! I have a question aboutCMD check. Athttps://bioconductor.org/checkResults/3.17/bioc-LATEST/a4Base/we see that thecheckpasses fora4Base. But when I runR CMD check .in a local clone I get: > > ~/bbs-3.17-bioc/R/bin/R CMD check . > * using log directory '/tmp/a4Base/..Rcheck' > * using R Under development (unstable) (2023-01-14 r83615) > * using platform: aarch64-unknown-linux-gnu (64-bit) > * R was compiled by > gcc (Ubuntu 11.3.0-1ubuntu1~22.04) 11.3.0 > GNU Fortran (Ubuntu 11.3.0-1ubuntu1~22.04) 11.3.0 > * running under: Ubuntu 22.04.1 LTS > * using session charset: UTF-8 > * checking for file './DESCRIPTION' ... ERROR > Required fields missing or empty: > 'Author' 'Maintainer' > * DONE > > Status: 1 ERROR > > and indeed there is neitherAuthornorMaintainerathttps://code.bioconductor.org/browse/a4Base/blob/RELEASE_3_16/DESCRIPTIONWhy the checks pass athttps://bioconductor.org/checkResults/3.17/bioc-LATEST/a4Base/?!

Martin Grigorov (03:41:36) (in thread): > I see why it workscontains a DESCRIPTION with aMaintainer. It seems the packaging process adds this field

Martin Morgan (04:19:51) (in thread): > Yes, check is meant to be run on the built package, R CMD build . creates the built package (e.g., on linux a4Base_1.46.0.tar.gz) and thenR CMD check a4Base_1.46.0.tar.gz

Martin Grigorov (04:23:02) (in thread): > Thanks,@Martin Morgan! I’ve just tested it andCMD build .indeed addsAuthorandMaintainer! All is fine!

2023-01-31

Aidan Lakshman (14:20:54): > Quick question: Is there any way to get (La)TeX code to display in.Rdfiles? I’m having an issue with multiline equations looking like garbage, and ideally I’d like to be able to use something similar to a\begin{align*} … \end{align*}environment. Unfortunately, I haven’t been able to find a way to make this work in documentation…only in.rmd/.qmdfiles, which doesn’t help for package building. > > For reference, the equation of interest (as I’d display it in latex) is: > > \begin{align*} > N(S) = (&n_A,n_C,n_G,n_T,\\ > &\;\mu_A,\mu_C,\mu_G,\mu_T,\\ > &\;D_2^A,D_2^C,D_2^G,D_2^T,\\ > &\;n_{AA},n_{AC},\dots,n_{TT},\\ > &\;n_{AAA},n_{AAC},\dots,n_{TTT}) > \end{align*} > > I’ll post photos of what I’m getting vs. wanting below.

Aidan Lakshman (14:22:16) (in thread): > this is what I’m getting–I’ve only been able to accomplish this using\deqn{} - File (PNG): Screen Shot 2023-01-31 at 14.22.00.png

Aidan Lakshman (14:24:48) (in thread): > here’s what I’d ideally like to see - File (PNG): Screen Shot 2023-01-31 at 14.24.36.png

Aidan Lakshman (14:27:50) (in thread): > I did manage to get something close by just putting a ton of\qquad\qquad\quad\;calls in the\deqn{}environment, but ideally there’d be a better solution…

Francesc Català-Moll (14:34:14): > @Francesc Català-Moll has joined the channel

Vince Carey (16:21:41) (in thread): > I’d suggest an R-help mailing list … there are knowledgeable people in bioc, but this is really for the R experts.

2023-02-01

Martin Morgan (10:52:35) (in thread): > Google took me tohttps://stackoverflow.com/a/37397807/547331from a knowledgeable poster and recent update pointing tomathjaxr(I think the challenge is to get the latex rendered in non-PDF formats…) - Attachment (Stack Overflow): Can R help manuals have latex math in them? > I am working on an R package and I am using the package Roxygen2 to write the help manuals for my R functions. What I would like to know is if it is possible to use latex for math equations in the …

2023-02-02

Aidan Lakshman (12:34:05) (in thread): > Yep, that’s what I had done previously, usingeqnanddeqn…seems like it’s not doable without external packages, which kinda sucks :/ > > GuessI’llstick to manually spacing it :p

2023-02-05

Martin Morgan (12:47:42): > For my package RedisParam the ‘devel’ build report shows failure on Linux and macOS because the CRAN dependencyreduxis not available. redux hasSystemDependency: hiredis; is redux failing to install because of this missing dependency? RedisParam is ok on all release builders. > > Also, is see on the RedisParamlanding pagethat there is a macOS arm64 binary available (great!), but I don’t see the macOS arm64 builder in the build report? - Attachment (Bioconductor): RedisParam (development version) > This package provides a Redis-based back-end for BiocParallel, enabling an alternative mechanism for distributed computation. The The ‘manager’ distributes tasks to a ‘worker’ pool through a central Redis server, rather than directly to workers as with other BiocParallel implementations. This means that the worker pool can change dynamically during job evaluation. All features of BiocParallel are supported, including reproducible random number streams, logging to the manager, and alternative ‘load balancing’ task distributions.

2023-02-06

Lori Shepherd (07:30:08): > The Arm64 build report is currently kept separate from the othershttps://bioconductor.org/checkResults/and is run on Tue and Friday for devel and Tue and Sun for release – I’ll let others investigate the system dependency to find out if it is or is not yet on – side note without really looking into it, if it is devel the mac binary for devel might not be available yet for the devel builder – but it would be odd that linux would be installed

Kasper D. Hansen (10:03:15): > That’s a useful page

Kasper D. Hansen (10:04:45): > I always go to the overview page (by which I mean this one):https://bioconductor.org/checkResults/3.17/bioc-LATEST/long-report.html. If it is easy to add, I think it could be useful to have a link tocheckResultsfrom the top of the overview page, where we have the different hosts listed

Martin Morgan (10:21:48) (in thread): > This seems like a useful addition to me, too. I look at the package-specific page (e.g.,https://bioconductor.org/checkResults/devel/bioc-LATEST/RedisParam/) linked from the package ‘landing page’https://bioconductor.org/pacakges/devel/RedisParam) by the ‘build’ button.

Vince Carey (11:58:53) (in thread): > Good to be getting some input on usage patterns for build reports. They are useful to us, with years of experience with this specific format, but can the key information be made even clearer?

Hervé Pagès (13:56:38) (in thread): > Adding the suggested link should be easy (@Andres Wokaty?). However please note that the current situation with the Arm64 builds (i.e. run twice a week only,R CMD checkskipped, report separated from the main daily report) is temporary only, until we have access to enough Arm64 computing power to run these builds daily and merge them into the main daily builds and report.

Andres Wokaty (14:00:41) (in thread): > https://github.com/Bioconductor/BBS/issues/262links for release and devel - Attachment: #262 Add link to checkResults on reports > Add link to checkResults on reports

Ying Chen (20:28:32): > Hi, I got a hub related question (not sure if here is the right place to put, but I will give it a try first), we are seeing BiocBuild error for all the platforms for our package bambu, which seems to be related to the hub cache again, as the attached message suggested, just wondering if someone could help out here? > > ############################################################################## > ############################################################################## > ### > ### Running command: > ### > ### /home/biocbuild/bbs-3.17-bioc/R/bin/R CMD build --keep-empty-dirs --no-resave-data bambu > ### > ############################################################################## > ############################################################################## > > > * checking for file 'bambu/DESCRIPTION' ... OK > * preparing 'bambu': > * checking DESCRIPTION meta-information ... OK > * cleaning src > * installing the package to build vignettes > * creating vignettes ... ERROR > --- re-building 'bambu.Rmd' using rmarkdown > [E::idx_find_and_load] Could not retrieve index file for '/home/biocbuild/.cache/R/ExperimentHub/1bcf8268327299_3845' > [E::idx_find_and_load] Could not retrieve index file for '/home/biocbuild/.cache/R/ExperimentHub/1bcf8268327299_3845' > [E::idx_find_and_load] Could not retrieve index file for '/home/biocbuild/.cache/R/ExperimentHub/1bcf8268327299_3845' > [E::idx_find_and_load] Could not retrieve index file for '/home/biocbuild/.cache/R/ExperimentHub/1bcf8268327299_3845' > [E::idx_find_and_load] Could not retrieve index file for '/home/biocbuild/.cache/R/ExperimentHub/1bcf8250b4e34e_3847' > [E::idx_find_and_load] Could not retrieve index file for '/home/biocbuild/.cache/R/ExperimentHub/1bcf826027621_3849' > [E::idx_find_and_load] Could not retrieve index file for '/home/biocbuild/.cache/R/ExperimentHub/1bcf822e9527d1_3851' > [E::idx_find_and_load] Could not retrieve index file for '/home/biocbuild/.cache/R/ExperimentHub/1bcf82afc9a27_3853' > [E::idx_find_and_load] Could not retrieve index file for '/home/biocbuild/.cache/R/ExperimentHub/1bcf8277be0001_3855' > [E::idx_find_and_load] Could not retrieve index file for '/home/biocbuild/.cache/R/ExperimentHub/1bcf8268327299_3845' > [E::idx_find_and_load] Could not retrieve index file for '/home/biocbuild/.cache/R/ExperimentHub/1bcf8250b4e34e_3847' > [E::idx_find_and_load] Could not retrieve index file for '/home/biocbuild/.cache/R/ExperimentHub/1bcf826027621_3849' > [E::idx_find_and_load] Could not retrieve index file for '/home/biocbuild/.cache/R/ExperimentHub/1bcf822e9527d1_3851' > [E::idx_find_and_load] Could not retrieve index file for '/home/biocbuild/.cache/R/ExperimentHub/1bcf82afc9a27_3853' > [E::idx_find_and_load] Could not retrieve index file for '/home/biocbuild/.cache/R/ExperimentHub/1bcf8277be0001_3855' > [E::idx_find_and_load] Could not retrieve index file for '/home/biocbuild/.cache/R/ExperimentHub/1bcf8268327299_3845' > [E::idx_find_and_load] Could not retrieve index file for '/home/biocbuild/.cache/R/ExperimentHub/1bcf8250b4e34e_3847' > [E::idx_find_and_load] Could not retrieve index file for '/home/biocbuild/.cache/R/ExperimentHub/1bcf826027621_3849' > [E::idx_find_and_load] Could not retrieve index file for '/home/biocbuild/.cache/R/ExperimentHub/1bcf822e9527d1_3851' > [E::idx_find_and_load] Could not retrieve index file for '/home/biocbuild/.cache/R/ExperimentHub/1bcf82afc9a27_3853' > [E::idx_find_and_load] Could not retrieve index file for '/home/biocbuild/.cache/R/ExperimentHub/1bcf8277be0001_3855' > [E::idx_find_and_load] Could not retrieve index file for '/home/biocbuild/.cache/R/ExperimentHub/1bcf8268327299_3845' > [E::idx_find_and_load] Could not retrieve index file for '/home/biocbuild/.cache/R/ExperimentHub/1bcf8250b4e34e_3847' > [E::idx_find_and_load] Could not retrieve index file for '/home/biocbuild/.cache/R/ExperimentHub/1bcf826027621_3849' > [E::idx_find_and_load] Could not retrieve index file for '/home/biocbuild/.cache/R/ExperimentHub/1bcf822e9527d1_3851' > [E::idx_find_and_load] Could not retrieve index file for '/home/biocbuild/.cache/R/ExperimentHub/1bcf82afc9a27_3853' > [E::idx_find_and_load] Could not retrieve index file for '/home/biocbuild/.cache/R/ExperimentHub/1bcf8277be0001_3855' > Quitting from lines 270-272 (bambu.Rmd) > Error: processing vignette 'bambu.Rmd' failed with diagnostics: > `...` must be empty. > ✖ Problematic argument: > • all = TRUE > --- failed re-building 'bambu.Rmd' > > SUMMARY: processing the following file failed: > 'bambu.Rmd' > > Error: Vignette re-building failed. > Execution halted >

Kasper D. Hansen (20:55:56): > The experts will chime in, but it happens that the cache gets corrupted on the build machines. This requires a manual intervention (to clear the cache) to fix.

Kasper D. Hansen (20:56:10): > This is probably what happened here

Kasper D. Hansen (20:56:58): > I would expect the best way to report this is to post on bioc-devel, although I believe all relevant people also monitor this slack channel.

Ying Chen (21:33:35): > thank you for the quick response!

Henrik Bengtsson (22:58:28): > Going to hijack this thread to ask a related question: I run into these type of errors “randomly” for Bioconductor packages when reverse-dependency checking my packages. Therevdepcheck::revdep_check()tool runs two check instances in parallel, one for the current version and one for the devel version. My current hypothesis when I see these errors is that theExperimentHub (BiocFileCache?) cache isnotparallel safe, i.e. it does not handle when multiple R processes try to update the same cache, e.g. one process writes to file just when another one tries to read from the same file. Does anyone know if this could be the case?

2023-02-07

Alan O’C (06:22:46): > Doesn’t look to be parallel safe, unless write_disk ishttps://code.bioconductor.org/browse/BiocFileCache/blob/RELEASE_3_16/R/httr.R?branch=RELEASE_3_16&file=R/httr.R#L44 - Attachment (code.bioconductor.org): Bioconductor Code: Browse > Browse the content of Bioconductor software packages.

Martin Morgan (08:26:24) (in thread): > The intention is that BiocFileCache be ‘parallel safe’. > > Lock access to the SQL database is implemented athttps://code.bioconductor.org/browse/BiocFileCache/blob/RELEASE_3_16/R/sql.R?branch=RELEASE_3_16&file=R/sql.R#L6, but it could be that the locks are not entirely robust to all workflows. > > It would be good to have a reproducible example… - Attachment (code.bioconductor.org): Bioconductor Code: Browse > Browse the content of Bioconductor software packages.

Henrik Bengtsson (12:51:28) (in thread): > Thanks for confirming it is designed to be parallel safe. To be clear, it’s rare, so not a common problem. Now when I know, I’ll keep an extra eye out for this and see if I can narrow in on a rep example. My revdep checks runs in parallel on the same host, so file-locking across hosts should not be involved (although it’s done to a shared, global BeeGFS file system).

Hervé Pagès (21:06:53) (in thread): > @Andres WokatyAlso for Martin’s original issue:https://github.com/Bioconductor/BBS/issues/264Thanks! - Attachment: #264 Accomodate CRAN package redux > @jwokaty > > Required by Bioconductor package RedisParam (Martin’s). > > • Linux builders: redux requires libhiredis-dev on Ubuntu: > > > > install.packages("redux") > trying URL '[https://ftp.osuosl.org/pub/cran/src/contrib/redux_1.1.3.tar.gz](https://ftp.osuosl.org/pub/cran/src/contrib/redux_1.1.3.tar.gz)' > Content type 'application/x-gzip' length 97908 bytes (95 KB) > ================================================== > downloaded 95 KB > > * installing **source** package ‘redux’ ... > **** package ‘redux’ successfully unpacked and MD5 sums checked > **** using staged installation > Using PKG_CFLAGS=-I/usr/include/hiredis > Using PKG_LIBS=-lhiredis > ------------------------- ANTICONF ERROR --------------------------- > Configuration failed because hiredis was not found. Try installing: > * deb: libhiredis-dev (Debian, Ubuntu, etc) > * rpm: hiredis-devel (Fedora, EPEL) > * brew: hiredis (OSX) > If hiredis is already installed, check that 'pkg-config' is in your > PATH and PKG_CONFIG_PATH contains a hiredis.pc file. If pkg-config > is unavailable you can set INCLUDE_DIR and LIB_DIR manually via: > R CMD INSTALL --configure-vars='INCLUDE_DIR=... LIB_DIR=...' > -------------------------------------------------------------------- > ERROR: configuration failed for package ‘redux’ > * removing ‘/home/hpages/R/R-4.3.r83438/library/redux’ > > The downloaded source packages are in > ‘/tmp/RtmpMvarDX/downloaded_packages’ > Updating HTML index of packages in '.Library' > Making 'packages.html' ... done > Warning message: > In install.packages("redux") : > installation of package ‘redux’ had non-zero exit status > > > • Windows builders: Nothing required there. CRAN always provides Windows binary packages for R release and devel (or at least they make a bin/windows/contrib/4.3 symlink that redirects to bin/windows/contrib/4.2). > • Mac builders (intel and arm64): CRAN provides no Mac binary packages (intel or arm64) for R devel at the moment so redux should join the list of difficult packages on this platform: https://github.com/Bioconductor/BBS/blob/master/Doc/Prepare-macOS-Mojave-HOWTO.md#what-if-cran-doesnt-provide-package-binaries-for-macos-yet|https://github.com/Bioconductor/BBS/blob/master/Doc/Prepare-macOS-Mojave-HOWTO.md#what-if-cran-doesnt-provide-package-binaries-for-macos-yet > > Thanks!

2023-02-08

Andres Wokaty (14:38:02): > @Martin Morgan@Kasper D. HansenI was talking to@Vince Careyabout adding links to the reports when he suggested it might be a good idea to discuss how the build reports are used to discover what’s useful or what’s needed. Would you be open to meeting about this? I’d also like to extend this to anyone who’s interested.

Kasper D. Hansen (14:38:39): > I am always happy to chat about this

2023-02-15

Andres Wokaty (14:31:49) (in thread): > I wasn’t able to get more interest in meeting about this. But if we could meet, I would appreciate your feedback and I could use that ask better questions from the community. Do you have any availability on Mondays or Fridays to meet for about 30 minutes just to talk about how you use the reports and what’s useful?

2023-03-07

Jonathan Griffiths (05:35:50): > A few packages seem to be having a citeproc issue on nebbiolo1 e.g. MGD. > > Error running filter pandoc-citeproc: > Could not find executable pandoc-citeproc > > Is this the right sort of place to report these issues?

Martin Grigorov (05:37:36) (in thread): > someone reported this on the dev mailing list yesterday. The core team is aware and afaik it should be fixed for the next build

Martin Grigorov (05:38:33) (in thread): > https://stat.ethz.ch/pipermail/bioc-devel/2023-March/019508.html

Jonathan Griffiths (05:38:54) (in thread): > thanks!

Jonathan Griffiths (05:40:22) (in thread): > bizarrely my gmail search didn’t pick up anything aboutpandoc-citeproc

Jonathan Griffiths (06:01:43) (in thread): > I see it now as the email has arrived: the perils of getting the daily digest.

2023-03-10

Joaquin Reyna (15:27:42): > @Joaquin Reyna has joined the channel

Edel Aron (15:34:45): > @Edel Aron has joined the channel

Anna Konstorum (16:34:56): > @Anna Konstorum has joined the channel

2023-03-13

Lori Shepherd (08:24:44): > The release schedule for Bioconductor release 3.17 has been announced. Please be mindful of important dates and deadlines.Bioconductor 3.17 Release Schedule

2023-03-15

Joaquin Reyna (13:29:45): > Hi Bioconductor Community, > > I am part of the nipalsMCIA team (https://github.com/Muunraker/nipalsMCIA) and we are trying to submit to Bioconductor (submission:https://github.com/Bioconductor/Contributions/issues/2914) however we’ve been running into a problem with aR CMD buildproblem that we haven’t been able to solve. Oddly enough our package will successfully build on our teams’ computers but not on Bioconductor (specifically themerida1host node). This mainly stems from the introduction of theMultiAssayExperimentinto our package and it seems to be a problem on the building phase rather thanMultiAssayExperimentitself. > > Thanks a lot for your help, > > nipalsMCIA Team@Joaquin Reyna@Anna Konstorum@Maximilian Mattessich@Edel Aron - Attachment: #2914 nipalsMCIA

Maximilian Mattessich (14:19:24): > @Maximilian Mattessich has joined the channel

Kasper D. Hansen (14:33:31): > This seems like a NAMESPACE issue > > **** testing if installed package can be loaded from temporary location > Warning: S3 method .DollarNames.MultiAssayExperiment was declared in NAMESPACE but not found > Error: package or namespace load failed for nipalsMCIA in namespaceExport(ns, exports): > undefined exports: experiments<-, sampleMap<-, ExperimentList, MatchedAssayExperiment, MultiAssayExperiment, MultiAssayExperimentToMAF, anyReplicated, experiments, exportClass, getWithColData, hasAssay, hasRowRanges, intersectColumns, intersectRows, listToMap, loadHDF5MultiAssayExperiment, longFormat, makeHitList, mapToList, mergeReplicates, prepMultiAssay, renameColname, renamePrimary, replicated, replicates, sampleMap, saveHDF5MultiAssayExperiment, subsetByAssay, subsetByColData, subsetByColumn, subsetByRow, upsetSamples, wideFormat > Error: loading failed >

Kasper D. Hansen (14:34:13): > But it is quite puzzling why this error is only present on merida1 and not on nebbiolo1

Kasper D. Hansen (14:37:27): > Its on the single package build, which I don’t know enough about. This really doesn’t seem like an issue which should be dependent on anything.

Kasper D. Hansen (14:37:58): > I was thinking if there is an issue with MultiAssayExperiment only building on one platform, but that doesn’t seem to be the case.

Joaquin Reyna (15:30:39) (in thread): > Yeah, it’s been a very difficult issue to debug for us.

Joaquin Reyna (15:32:04) (in thread): > We started going down the rabbit hole before we found this Bioconductor Slack group.

Kasper D. Hansen (16:18:48) (in thread): > The other place is the devel email list

Kasper D. Hansen (16:19:21) (in thread): > I don’t see a post about this on that email list and that would be the canonical place to ask

Joaquin Reyna (16:19:44) (in thread): > Thanks Kasper! I’ll shoot a message there.

Kasper D. Hansen (16:19:55) (in thread): > You might have equally good luck here

Joaquin Reyna (16:20:03) (in thread): > But I do appreciate you taking a look

Kasper D. Hansen (16:20:16) (in thread): > I am just saying that I would have expected you to stumble upon the devel email list first

Joaquin Reyna (16:20:27) (in thread): > I see

Kasper D. Hansen (16:20:34) (in thread): > The people who can help reads both the email list and this channel

Kasper D. Hansen (16:20:48) (in thread): > So I don’t think you’ll have better luck there

Joaquin Reyna (16:22:06) (in thread): > How are these issues typically resolved?

Joaquin Reyna (16:22:36) (in thread): > Does the owner ofmerida1have to look into this?

Kasper D. Hansen (16:23:25) (in thread): > The core team will probably reply later. I have extensive experience with the build system and I think it looks weird.

Joaquin Reyna (16:23:27) (in thread): > It’s our first time making a Bioconductor package so we’re not really experienced with all of these types of issues.

Kasper D. Hansen (16:24:21) (in thread): > There are usually many ways things can get messed up with accidently running different code etc., but in my understanding of the single package build system, that should not be an issue.

Joaquin Reyna (16:25:31) (in thread): > So you would recommend just waiting until a core team member replies?

Joaquin Reyna (16:25:41) (in thread): > I just don’t know how to debug from our side

Kasper D. Hansen (16:27:15) (in thread): > Yes, I would wait. Things occasionally break in the build system, and usually I would suspect that you made a mistake, but in this case it looks weird

Kasper D. Hansen (16:27:32) (in thread): > However, there is nothing preventing you from fixing the other warnings and notes

Joaquin Reyna (16:33:53) (in thread): > Thanks a lot for your suggestions. We’ve been trying to decide where to direct our efforts so for now we’ll just wait a bit more.

Kasper D. Hansen (16:34:54) (in thread): > I don’t understand. You have to fix these other things anyway. Why are you waiting?

Joaquin Reyna (16:37:46) (in thread): > I’m not.

Joaquin Reyna (16:38:05) (in thread): > We are working on what we have to work on.

2023-03-16

Kasper D. Hansen (15:25:21): > @Lori Shepherd@Hervé PagèsNot sure if you’re in this particular channel or whether you’re the right people, but take a look at@Joaquin Reyna’s issue. I have a quick analysis in my reply.

Vince Carey (15:44:35): > Thanks@Kasper D. Hansenit is being investigated.

Kasper D. Hansen (15:47:20): > Great. I guess I am getting used to extremely rapid responses, so I was thinking it didn’t get noticed

Andres Wokaty (16:12:55): > @Joaquin ReynaI was looking into this issue and was able to manually build your package on merida1 without error: > > merida1:tmp biocbuild$ R CMD build nipalsMCIA > * checking for file 'nipalsMCIA/DESCRIPTION' ... OK > * preparing 'nipalsMCIA': > * checking DESCRIPTION meta-information ... OK > * installing the package to build vignettes > * creating vignettes ... OK > * checking for LF line-endings in source and make files and shell scripts > * checking for empty or unneeded directories > * looking to see if a 'data/datalist' file should be added > * building 'nipalsMCIA_0.99.1.tar.gz' > > I’m going to kick off the single package builder on your machine again tomorrow when Lori is around. If we get the same issue, my recommendation will be to proceed with the review without merida1 and focus on linux build with the caveat that there still might potentially be problems to work out after your package is added to the daily builds.

Anna Konstorum (20:26:25) (in thread): > Thank you very much for looking into this@Andres Wokaty!@Joaquin Reyna, myself, and the rest of thenipalsMCIAteam (@Edel Aron,@Maximilian Mattessich) appreciate your help. Please let us know if there’s anything we can do on our end to help address this issue.

2023-03-17

Lori Shepherd (08:12:42) (in thread): > The build ERROR namespace issue has been resolved. Likely it occurred because of unlucky timing where the package was being reinstalled on our daily builder. The SPB shares the same R and R library with the daily builder to avoid redundancy and to save space so we occasionally see these unlucky clashes.

Joaquin Reyna (13:34:33) (in thread): > Thanks a lot! I’m glad to hear it was just a minor issue.

2023-03-20

Steffen Neumann (05:26:34): > Hi, over the weekend the mzR build on merida1 was unhappy due to an issue with the ncdf4 dependency

Steffen Neumann (05:27:13) (in thread): > http://bioconductor.org/checkResults/devel/bioc-LATEST/mzR/merida1-install.htmlhas > > **** byte-compile and prepare package for lazy loading > Error in dyn.load(file, DLLpath = DLLpath, ...) : > unable to load shared object '/Library/Frameworks/R.framework/Versions/4.3/Resources/library/ncdf4/libs/ncdf4.so': > dlopen(/Library/Frameworks/R.framework/Versions/4.3/Resources/library/ncdf4/libs/ncdf4.so, 6): Library not loaded: /Library/Frameworks/R.framework/Versions/4.2/Resources/lib/libR.dylib > Referenced from: /Library/Frameworks/R.framework/Versions/4.3/Resources/library/ncdf4/libs/ncdf4.so > Reason: image not found >

Steffen Neumann (05:28:11) (in thread): > Usually these disappear after a few days when the BioC fairies have sorted things out, so I’ll patiently lean back for now:slightly_smiling_face:

Johannes Rainer (07:17:55): > @Johannes Rainer has joined the channel

Vince Carey (07:50:00) (in thread): > I had no trouble on my intel mac with ncdf or mzR from source with today’s R-devel. Hope this sorts out.

2023-03-21

Hervé Pagès (17:10:32) (in thread): > Probably related tohttps://stat.ethz.ch/pipermail/bioc-devel/2023-March/019522.html, which it seems Jen has fixed:https://bioconductor.org/checkResults/3.17/bioc-LATEST/mzR/

2023-03-23

Martin Grigorov (04:29:13): > Hello! Is there a setting/envvar that controls the timeout for downloading a package ? e.g.BiocManager::install(c("BSgenome.Hsapiens.UCSC.hg19"), type = "source", checkBuilt = TRUE)

Martin Grigorov (04:29:31) (in thread): > > Error in download.file(url, destfile, method, mode = "wb", ...) : > download from '[https://bioconductor.org/packages/3.17/data/annotation/src/contrib/BSgenome.Hsapiens.UCSC.hg19_1.4.3.tar.gz](https://bioconductor.org/packages/3.17/data/annotation/src/contrib/BSgenome.Hsapiens.UCSC.hg19_1.4.3.tar.gz)' failed > In addition: Warning messages: > 1: In download.file(url, destfile, method, mode = "wb", ...) : > downloaded length 5368359121760235870414891528490663964317552158853786016275609710822194178695399138893332719749836647836800813529008606250955986594182807812773770643717048831262137586111765449802483529491827008662386243580906798949335727840241334432659215154402243379200 != reported length 0 > 2: In download.file(url, destfile, method, mode = "wb", ...) : > URL '[https://bioconductor.org/packages/3.17/data/annotation/src/contrib/BSgenome.Hsapiens.UCSC.hg19_1.4.3.tar.gz](https://bioconductor.org/packages/3.17/data/annotation/src/contrib/BSgenome.Hsapiens.UCSC.hg19_1.4.3.tar.gz)': Timeout of 300 seconds was reached > Warning in download.packages(pkgs, destdir = tmpd, available = available, : > download of package 'BSgenome.Hsapiens.UCSC.hg19' failed >

Martin Grigorov (04:30:27) (in thread): > I tried withexport R_DEFAULT_INTERNET_TIMEOUT=60, to see whether it will fail earlier but it failed again with300

Martin Grigorov (05:06:52) (in thread): > I found it!R_ENVIRON_USER=~/BBS/3.17/Renviron.bioc /home/biocbuild/bbs-3.17-bioc/R/bin/Rworks!

Martin Grigorov (05:09:00) (in thread): > @Hervé PagèsIs ~/BBS/3.17/Renviron.bioc loaded forrun.sh? I aks because some reports fail with timeout for Linux ARM64 > Recently we updated R-devel to a newer version and this caused some troubles:confused:

Martin Grigorov (05:24:42) (in thread): > It is also possible that the network was slow for another reason during run.sh and even 2400secs was not enough …

Martin Morgan (07:26:27) (in thread): > I guess an .Renviron would override your use of an environment variable, but from?download.file > > The timeout for many parts of the transfer can be set by the > > option ‘timeout’ which defaults to 60 seconds. This is often > > insufficient for downloads of large files (50MB or more) and so > > should be increased when ‘download.file’ is used in packages to do > > so. Note that the user can set the default timeout by the > > environment variable ‘R_DEFAULT_INTERNET_TIMEOUT’ in recent > > versions of R, so to ensure that this is not decreased packages > > should use something like > > > > options(timeout = max(300, getOption(“timeout”))) > > > > (It is unrealistic to require download times of less than 1s/MB.)

Martin Grigorov (07:27:42) (in thread): > Using the env var didn’t help here. I guess something else in BBS overwrites it

Martin Grigorov (07:28:37) (in thread): > But providing custom R_ENVIRON_USER helped (https://github.com/Bioconductor/BBS/blob/devel/3.17/Renviron.bioc#L41)

Yikun Jiang (10:33:40) (in thread): > https://github.com/Bioconductor/BBS/pull/255#issuecomment-1418551472

Yikun Jiang (10:33:43) (in thread): > this might help

Hervé Pagès (13:34:53) (in thread): > Right, already discussed last month at the link provided by@Yikun Jiang. Thanks@Yikun Jiang

Martin Grigorov (15:10:31) (in thread): > The thing is that it does not download the data packages during run.sh. We install them manually

Martin Grigorov (15:13:55) (in thread): > I see nothing in the logs about timeouts. The errors are that some dependencies are not installed. Installing them manually works only if I use R_ENVIRON_USER. Exporting the suggested env vars did not change the timeout duration

Hervé Pagès (16:51:18) (in thread): > The logs in~/bbs-3.17-bioc/logwon’t report any package specific installation problems, you want to look at the*.install-out.txtfiles in~/public_html/BBS/3.17/bioc/products-in/<builder-name>/install/for that. If you see a download.file() timeout there then you need to increaseR_DEFAULT_INTERNET_TIMEOUTin~/BBS/3.17/Renviron.bioc. That won’t be effective until you run therun.shscript again though.

2023-03-24

Martin Grigorov (02:46:59) (in thread): > Thanks!

2023-03-27

koki (07:52:52): > Hi, my packagescTGIF(version 1.13.0) is currently failing on the Bioconductor devel branch (errors on macOS).https://bioconductor.org/checkResults/3.17/bioc-LATEST/scTGIF/I’m not familar with the error “Error: Graphics API version mismatch” but I suspect this is due to the server side rather than the source code ofscTGIF. > In the following post, it seems that reinstallingCairo,naniar, andraggimproved such a situation.https://stackoverflow.com/questions/68753250/getting-the-error-graphics-api-version-mismatchCould someone from the core team look into this? - Attachment (Stack Overflow): Getting the Error: Graphics API version mismatch > I’m getting the following error when I run shiny: > Error: Graphics API version mismatch > Listening on http://127.0.0.1:3774 > Warning: Error in Cairo: Graphics API version mismatch > [No stack trace

Lori Shepherd (09:22:00) (in thread): > @Andres Wokatycould you look into this – it was also reported on the bioc-devel mailing list with the suggestion of reinstalling Cairo too.

Andres Wokaty (09:24:31) (in thread): > I am working on this. I had to reinstallragg. I am just verifying it’s resolved, but these errors should be cleared up on tomorrow’s report.

Andres Wokaty (16:11:26): > On March 29, we’ll add_R_CHECK_SUGGESTS_ONLY=trueto our devel 3.17 Linux builder to detect missing package dependencies. You can read more about the motivation behind the issue athttps://github.com/Bioconductor/BBS/issues/248. > > If you notice after March 29 that one of your packages is failing R CMD check, consider adding this line to your Renviron.bioc. To reproduce the error, you will need to have a site-library directory in your R home directory, so you’ll have to reinstall R and make thesite-librarybefore you see the error. > > To resolve the error, you should identify the missing dependency and add it to Suggests in the DESCRIPTION file. - Attachment: #248 WISH: Check with *R_CHECK_SUGGESTS_ONLY*=true to identify missing package dependencies > (This replaces invalid feature requests #242, #243, and #244.) > > Background > > Using *R_CHECK_SUGGESTS_ONLY*=true will run R CMD check with sandboxed an R library, where on declared package dependencies in DESCRIPTION + recursive hard dependencies are included. If an example, package test, or a vignette needs a package that is not in this set of dependencies, it will not be available to be loaded and result in an error. > > Wish > > Add *R_CHECK_SUGGESTS_ONLY*=true to Renviron.bioc to detect missing package dependencies. > > R CMD check sets up a sandboxed R library for it’s life span. It <https://github.com/wch/r-source/blob/8bf881e5da4da60f3aaad1b1812924817906bb29/src/library/tools/R/check.R#L138-L277|uses symbolic file links to do this>, so the performance overhead for doing this is really small. I could not detect a performance penalty (see below for details). > > Examples > > I ran the following examples using Renviron.bioc for Bioconductor 3.17: > > > wget [https://raw.githubusercontent.com/Bioconductor/BBS/master/3.17/Renviron.bioc](https://raw.githubusercontent.com/Bioconductor/BBS/master/3.17/Renviron.bioc) > > > I also ran with: > > > export *R_CHECK_TESTS_NLINES*=17 > > > to make it clear what one of the ERRORs is about. > > Example 1: Package bnem (missing RUnit for tests and BiocStyle for vignettes) > > Downloading bnem package source tarball: > > > $ wget [https://bioconductor.org/packages/devel/bioc/src/contrib/bnem_1.7.0.tar.gz](https://bioconductor.org/packages/devel/bioc/src/contrib/bnem_1.7.0.tar.gz) > > > Current Bioconductor 3.17 checks > > When checking with: > > > $ R_CHECK_ENVIRON=Renviron.bioc R --vanilla CMD check bnem_1.7.0.tar.gz > > > package passes with a single NOTE. > > * using log directory ‘/tmp/bioc/bnem.Rcheck’ > * using R Under development (unstable) (2022-11-30 r83391) > * using platform: x86_64-pc-linux-gnu (64-bit) > * using session charset: UTF-8 > * checking for file ‘bnem/DESCRIPTION’ … OK > * checking extension type … Package > * this is package ‘bnem’ version ‘1.7.0’ > * package encoding: UTF-8 > * checking package namespace information … OK > * checking package dependencies … OK > * checking if this is a source package … OK > * checking if there is a namespace … OK > * checking for hidden files and directories … OK > * checking for portable file names … OK > * checking for sufficient/correct file permissions … OK > * checking whether package ‘bnem’ can be installed … OK > * checking installed package size … OK > * checking package directory … OK > * checking ‘build’ directory … OK > * checking DESCRIPTION meta-information … OK > * checking top-level files … OK > * checking for left-over files … OK > * checking index information … OK > * checking package subdirectories … OK > * checking R files for non-ASCII characters … OK > * checking R files for syntax errors … OK > * checking whether the package can be loaded … OK > * checking whether the package can be loaded with stated dependencies … OK > * checking whether the package can be unloaded cleanly … OK > * checking whether the namespace can be loaded with stated dependencies … OK > * checking whether the namespace can be unloaded cleanly … OK > * checking loading without being on the library search path … OK > * checking dependencies in R code … NOTE > Namespace in Imports field not imported from: ‘rmarkdown’ > All declared Imports should be used. > * checking S3 generic/method consistency … OK > * checking replacement functions … OK > * checking foreign function calls … OK > * checking R code for possible problems … OK > * checking Rd files … OK > * checking Rd metadata … OK > * checking Rd cross-references … OK > * checking for missing documentation entries … OK > * checking for code/documentation mismatches … OK > * checking Rd sections … OK > * checking Rd contents … OK > * checking for unstated dependencies in examples … OK > * checking contents of ‘data’ directory … OK > * checking data for non-ASCII characters … OK > * checking data for ASCII and uncompressed saves … OK > * checking installed files from ‘inst/doc’ … OK > * checking files in ‘vignettes’ … OK > * checking examples … OK > * checking for unstated dependencies in ‘tests’ … OK > * checking tests … > Running ‘runTests.R’ > OK > * checking for unstated dependencies in vignettes … OK > * checking package vignettes in ‘inst/doc’ … OK > * checking running R code from vignettes … > ‘bnem.rmd’ using ‘UTF-8’… OK > NONE > * checking re-building of vignette outputs … OK > * checking PDF version of manual … OK > * DONE > > Status: 1 NOTE
> See
> ‘/tmp/bioc/bnem.Rcheck/00check.log’
> for details. > With also *R_CHECK_SUGGESTS_ONLY*=true > > When checking with: > > > $ *R_CHECK_SUGGESTS_ONLY*=true R_CHECK_ENVIRON=Renviron.bioc R --vanilla CMD check bnem_1.7.0.tar.gz > > > the package fails with two ERRORs: > > > * checking tests ... > Running ‘runTests.R’ > ERROR > Running the tests in ‘tests/runTests.R’ failed. > Complete output: > > BiocGenerics:::testPackage("bnem") > Error in library("RUnit", quietly = TRUE) : > there is no package called 'RUnit' > > > and > > > * checking re-building of vignette outputs ... ERROR > Error(s) in re-building vignettes: > ... > --- re-building ‘bnem.rmd’ using rmarkdown > Error: processing vignette 'bnem.rmd' failed with diagnostics: > there is no package called ‘BiocStyle’ > > > * using log directory ‘/tmp/bioc2/bnem.Rcheck’ > * using R Under development (unstable) (2022-11-30 r83391) > * using platform: x86_64-pc-linux-gnu (64-bit) > * using session charset: UTF-8 > * checking for file ‘bnem/DESCRIPTION’ … OK > * checking extension type … Package > * this is package ‘bnem’ version ‘1.7.0’ > * package encoding: UTF-8 > * checking package namespace information … OK > * checking package dependencies … OK > * checking if this is a source package … OK > * checking if there is a namespace … OK > * checking for executable files … OK > * checking for hidden files and directories … OK > * checking for portable file names … OK > * checking for sufficient/correct file permissions … OK > * checking whether package ‘bnem’ can be installed … OK > * checking installed package size … OK > * checking package directory … OK > * checking ‘build’ directory … OK > * checking DESCRIPTION meta-information … OK > * checking top-level files … OK > * checking for left-over files … OK > * checking index information … OK > * checking package subdirectories … OK > * checking R files for non-ASCII characters … OK > * checking R files for syntax errors … OK > * checking whether the package can be loaded … OK > * checking whether the package can be loaded with stated dependencies … OK > * checking whether the package can be unloaded cleanly … OK > * checking whether the namespace can be loaded with stated dependencies … OK > * checking whether the namespace can be unloaded cleanly … OK > * checking loading without being on the library search path … OK > * checking dependencies in R code … NOTE > Namespace in Imports field not imported from: ‘rmarkdown’ > All declared Imports should be used. > * checking S3 generic/method consistency … OK > * checking replacement functions … OK > * checking foreign function calls … OK > * checking R code for possible problems … OK > * checking Rd files … OK > * checking Rd metadata … OK > * checking Rd cross-references … OK > * checking for missing documentation entries … OK > * checking for code/documentation mismatches … OK > * checking Rd sections … OK > * checking Rd contents … OK > * checking for unstated dependencies in examples … OK > * checking contents of ‘data’ directory … OK > * checking data for non-ASCII characters … OK > * checking data for ASCII and uncompressed saves … OK > * checking installed files from ‘inst/doc’ … OK > * checking files in ‘vignettes’ … OK > * checking examples … OK > * checking for unstated dependencies in ‘tests’ … OK > * checking tests … > Running ‘runTests.R’ > ERROR > Running the tests in ‘tests/runTests.R’ fa…

2023-03-29

Matteo Tiberti (10:11:58): > @Matteo Tiberti has joined the channel

Himel Mallick (16:56:33): > @Himel Mallick has joined the channel

2023-04-03

Mengbo Li (20:45:15): > @Mengbo Li has joined the channel

2023-04-04

Jacques SERIZAY (05:03:13): > @Jacques SERIZAY has joined the channel

Jacques SERIZAY (06:54:54): > Hi all, > TheHiCoolbuild returns anexpectedWindows-specific (palomino3)buildERROR:https://bioconductor.org/checkResults/devel/bioc-LATEST/HiCool/palomino3-buildsrc.html. This is expected becauseHiCoolattempts to create abasilisk-managedcondaenvironment which includessamtoolsandbowtie2, which do not have recipes for Windows. This means the package cannot properly work on Windows machines. Any suggestion on how to deal with this? How does this impact HiCool being hosted by Bioconductor?

Lori Shepherd (07:30:04) (in thread): > In the top level of your directory include a.BBSoptionshidden file. In it specify what platform should be excluded that your package cannot handle with an UnsupportedPlatforms variableUnsupportedPlatforms: win

Mike Smith (09:45:59) (in thread): > Could you use theRsamtoolsandRbowtie2packages? Both of those are supported on Windows. - Attachment (Bioconductor): Rsamtools > This package provides an interface to the ‘samtools’, ‘bcftools’, and ‘tabix’ utilities for manipulating SAM (Sequence Alignment / Map), FASTA, binary variant call (BCF) and compressed indexed tab-delimited (tabix) files. - Attachment (Bioconductor): Rbowtie2 > This package provides an R wrapper of the popular bowtie2 sequencing reads aligner and AdapterRemoval, a convenient tool for rapid adapter trimming, identification, and read merging. The package contains wrapper functions that allow for genome indexing and alignment to those indexes. The package also allows for the creation of .bam files via Rsamtools.

Jacques SERIZAY (10:36:52) (in thread): > I considered it, however these dependencies are required by the python package I myself need (which is why I initially relied onbasilisk), so sadly it isn’t really an option:unamused:

2023-04-05

Josué Curto Navarro (20:59:39): > @Josué Curto Navarro has joined the channel

2023-04-06

Steffen Neumann (04:19:32): > Hi, since, um, maybe March 29, I have a build error related to something to do with dependencies in CAMERA. However, only for ONE of the architectures.

Steffen Neumann (04:22:23) (in thread): > http://bioconductor.org/checkResults/devel/bioc-LATEST/CAMERA/nebbiolo1-checksrc.htmlSpecifically, CAMERA calls an XCMS function, which errors becauserequireNamespace("multtest")inside xcmsfails. If this were related to theR_CHECK_SUGGESTS_ONLY, shouldn’t that fail on all arches now ?! > Tracking inhttps://github.com/sneumann/xcms/issues/671 - Attachment: #671 Handling multtest dependency (BioC build error) > Hi, I think we have leftover historic misuse in our dependency handling. > > In the old days, we avoided bloating dependencies, because some default behaviour was to recursively install all Suggested packages, and e.g. multtest pulled in a lot of that. So, we have > > https://github.com/sneumann/xcms/blob/fcab3c6b00de176fd7613f6c290f0b2469756af7/R/methods-xcmsSet.R#L1565|xcms/R/methods-xcmsSet.R > > Line 1565 in fcab3c6 > > Which in turn triggers a build error on BioC builds with new R-devel (r83996)
> Error report on https://master.bioconductor.org/checkResults/3.17/bioc-LATEST/CAMERA/nebbiolo1-checksrc.html > > > Test files with failing tests: > test_annotateDiffreport.R > test.annotateDiffreport.calcCaS > test.annotateDiffreport.calcCiS > test.annotateDiffreport.calcIso > test.annotateDiffreport.graphMethod > test.annotateDiffreport.intval > test.annotateDiffreport.pval > ... > Error in .local(object, ...) : > The use of 'diffreport' requires package 'multtest'. Please install with 'Biobase::install("multtest")' >

Vince Carey (06:14:37) (in thread): > I don’t think so. The reasons are technical and your hypothesis may be correct in the future. Others on team can give more details. But at the moment I think the main reporter of SUGGESTS_ONLY events is the linux builder.

Lori Shepherd (09:01:39) (in thread): > Yeswithout going into the technical,it’sonly active on Linux right now.

Hervé Pagès (10:26:37) (in thread): > See‘Note: If “R CMD check” recently failed on the Linux builder over a missing dependency, …’in the dotted box at the top of the report.

Steffen Neumann (10:41:30) (in thread): > Yes, saw that. Feels wrong that I have to add the dependencies of my dependency to my suggests. I was hoping for a suggestion for a fix to xcms resolving the issue in camera

Henrik Bengtsson (11:33:20) (in thread): > A package may choose to depend on another package “softly”, i.e. put it underSuggests:Sometimes it’s because it’s only needed duringR CMD buildandR CMD check, but not during install or at run-time. Sometimes it’s because it doesn’t want to bring in a full heavy dependency graph all the time, e.g. it might only be used at very rare occasions by a very small number of users. Other times it might be because a package is only available on one platform, and sometimes it’s a very-hard-to-install package. > > If you indirectly depend on such a soft dependency, then you’ll need to declare that you are indeed going to use it in your package (softly or not), e.g. inSuggests:orImports:.

Henrik Bengtsson (12:31:03) (in thread): > (FWIW, theR_CHECK_SUGGESTS_ONLY=truechecks were added to detect this type of direct and indirect undeclared dependencies)

2023-04-10

Lori Shepherd (08:22:11): > reminder: we are on schedule for our release as given inhttp://bioconductor.org/developers/release-schedule/and as of noon EDT today, we will be freezing the RELEASE_3_16 branch. We will be running the last builds for Release 3.16 over the next few days (tonight for software, tomorrow for data experiment, etc… per the schedule:https://bioconductor.org/checkResults/) . After noon today, no changes will be allowed to the RELEASE_3_16 branch ever; this includes bug fixes and hot fixes. As clarification, you will still be able to commit to the devel branch and in two weeks when we officially release the core team will create a RELEASE_3_17 branch for all packages (Do NOT try to create this branch yourself on thegit.bioconductor.orgserver).

Lori Shepherd (11:14:25): > List of packages scheduled to be deprecated in 3.17 and removed in 3.18:https://support.bioconductor.org/p/9150754/

Lori Shepherd (12:01:37): > The RELEASE_3_16 branch is now frozen to all push commits

2023-04-12

Nils Hoffmann (11:18:14): > @Nils Hoffmann has joined the channel

2023-04-17

Johannes Rainer (03:53:12): > I’m seeing some strange errors on macOS for Bioc 3.17 devel build report from 2023-04-14: > > example forrols:https://bioconductor.org/checkResults/3.17/bioc-LATEST/rols/merida1-buildsrc.html > > --- re-building 'rols.Rmd' using rmarkdown > Error: processing vignette 'rols.Rmd' failed with diagnostics: > The specified directory 'rols.Rmd' does not exist. > Warning in file(con, "r") : > cannot open file 'rols.Rmd': No such file or directory > Error: tangling vignette 'rols.Rmd' failed with diagnostics: > cannot open the connection > --- failed re-building 'rols.Rmd' > > OrROseq:https://bioconductor.org/checkResults/3.17/bioc-LATEST/ROSeq/merida1-buildsrc.html > > * creating vignettes ... ERROR > --- re-building 'ROSeq.Rmd' using rmarkdown > Warning in file(con, "r") : > cannot open file 'ROSeq.Rmd': No such file or directory > Error: tangling vignette 'ROSeq.Rmd' failed with diagnostics: > cannot open the connection > --- failed re-building 'ROSeq.Rmd' > > orRTN:https://bioconductor.org/checkResults/3.17/bioc-LATEST/RTN/merida1-buildsrc.html > > * creating vignettes ... ERROR > --- re-building 'RTN.Rmd' using rmarkdown > Error: processing vignette 'RTN.Rmd' failed with diagnostics: > The specified directory 'RTN.Rmd' does not exist. > Warning in file(con, "r") : > cannot open file 'RTN.Rmd': No such file or directory > Error: tangling vignette 'RTN.Rmd' failed with diagnostics: > cannot open the connection > --- failed re-building 'RTN.Rmd' > > seems to be the same/similar theme to these (might also be more package affected, but I did not check all). To me this looks maybe like some issue with the macOS builder or the R version used there (since the error does not happen on any other platform)?

Andres Wokaty (09:14:36) (in thread): > Thanks for bring this up. I’ll look at these errors. I’ve been working to update merida1 for the past week.

Andres Wokaty (11:19:51) (in thread): > These errors appear to be resolved in today’s build report.

2023-04-18

Alan Murphy (11:22:01): > Hey, I’m getting a strange build error for MSS (https://bioconductor.org/checkResults/3.17/bioc-LATEST/MungeSumstats/merida1-checksrc.html) over the past few days where a unit test is failing on mac (but today it is failing on Windows too for the first time). I can’t seem to replicate the issue locally on mac with R 4.3.0 (Intel mac) and Bioconductor version 3.17 or on the bioconductor dev docker image although this is linux which isn’t failing for this reason (there is a package missing on the linux machine today but it was passing previously)

Jermiah Joseph (16:41:32): > @Jermiah Joseph has joined the channel

Jermiah Joseph (16:58:24): > Hi All, thepianopackage is not available for my package dependency. How can I fix this?http://bioconductor.org/checkResults/devel/bioc-LATEST/CoreGx/**** installing to library ‘F:/biocbuild/bbs-3.17-bioc/R/library’*ERROR: dependency ‘piano’ is not available for package ‘CoreGx’

Andres Wokaty (17:13:29): > @Jermiah JosephThis might be because I updated R on the devel builders to R RC yesterday. Usually the builders need a couple of days to reinstall everything. If it’s not resolved by the end of the week, I can look into it.

2023-04-19

Johannes Rainer (01:18:11) (in thread): > Perfect! thanks!

Jacques SERIZAY (08:57:28) (in thread): > Following up on this 3 weeks later: I used to have the same error forHiContacts, and it was fixed when@Andres Wokatyreinstalledragg. Linux Bioconductor builds forHiContactshave been successful since then (https://github.com/js2264/HiContacts/actions/runs/4742906941/jobs/8421760921). However, I am also checking my commits withrworkflows-based GHA, which relies on the Bioconductor docker image, and the GitHub Linux build still fails with the sameGraphics APIissue (https://github.com/js2264/HiContacts/actions/runs/4742906941/jobs/8421760921#step:4:5758) (On Github build,ragg 1.2.5(latest) is installed). I would have thought that using a docker image would dodge this type of issue:thinking_face:. Any idea on how to fix this?

Jermiah Joseph (11:13:20) (in thread): > Im a little concerned about the deadline of this friday for packages passing all build and install tests without errors but if the piano fails on the windows platform I might not be able to make sure it wont produce errors by the friday deadline. Is there anything I can do to make sure that doesnt happen if I dont have access to windows machines?

Andres Wokaty (11:30:40) (in thread): > CoreGx is passing on today’s report.

Federico Marini (14:01:47): > @Andres Wokatyany idea of why this fails but only on one platform?https://master.bioconductor.org/checkResults/3.17/bioc-LATEST/pcaExplorer/nebbiolo1-checksrc.html

Federico Marini (14:01:58): > also goes for theidealpackage

Federico Marini (14:02:46): > “Enforcing”markdownas a dependency would be my best guess, but I dont get why it does not fail on all systems then..

Henrik Bengtsson (14:28:48) (in thread): > Seehttps://community-bioc.slack.com/archives/CEQ04GKEC/p1679947886937889 - Attachment: Attachment > On March 29, we’ll add _R_CHECK_SUGGESTS_ONLY=true to our devel 3.17 Linux builder to detect missing package dependencies. You can read more about the motivation behind the issue at https://github.com/Bioconductor/BBS/issues/248. > > If you notice after March 29 that one of your packages is failing R CMD check, consider adding this line to your Renviron.bioc. To reproduce the error, you will need to have a site-library directory in your R home directory, so you’ll have to reinstall R and make the site-library before you see the error. > > To resolve the error, you should identify the missing dependency and add it to Suggests in the DESCRIPTION file.

Federico Marini (14:30:55) (in thread): > Thanks@Henrik BengtssonI did miss that, my bad

Y-h. Taguchi (20:06:10): > @Y-h. Taguchi has joined the channel

Y-h. Taguchi (20:06:48): > https://master.bioconductor.org/checkResults/3.17/bioc-LATEST/TDbasedUFEadv/nebbiolo1-checksrc.htmlThanks for your every effort to maintain bioconductor. > I am so sorry, but TDbasedUEEadv had a problem again. Since I did not commit since the last successful build, the problem seems to be not in my side, but in the system. > I am glad if you can check the problem in the eraliest opportunity and fix it,

Hervé Pagès (21:36:08) (in thread): > We’ve already seen these timeouts before forTDbasedUFEadv. They seem intermittent. I’m not sure it’s on our side either. They’re hard to reproduce but maybe if you runR CMD checkon the package enough times (on Linux), you would be able to get one too, who knows. Does the package or one of the packages it depends on try to access remote resources?

Y-h. Taguchi (21:55:47) (in thread): > @Hervé PagèsI understand. Thus it might not be the problem in your side, but in my side. > > Does the package or one of the packages it depends on try to access remote resources? > Yes, it does. It might be the cause of intermittent time out. Should not I include the package to perform remote access in the internet?

Hervé Pagès (22:07:49) (in thread): > What is this remote resource? It’s hard to give advice when we have almost zero information on what’s going on. If accessing this remote resource is for downloading some data that you use in the examples or vignette, you should absolutely consider caching this data. Also what’s the size of this data? Are you always accessing the same dataset? > Otherwise, think of how it will impact your users if the remote resource cannot be accessed reliably. Have you reported the intermittent access problems to the people that take care of this resource? Do they shut it down on a regular basis for maintenance reasons?

Y-h. Taguchi (22:12:52) (in thread): > @Hervé PagèsThank you very much for your comments. Although it does have access to internet, I think that the problem is > you can see > > EllapsedTime: 2400.0 secondsRetCode: NoneStatus:**** TIMEOUT** > This has happened previously because of lack of -no-vignettes options. > Last time this has happened, the problem is resolved without the effort in my side. 2400 seconds was not reproduced in local test by core team. Is it possible for you to check if it can happen in your local machine? - File (PNG): image.png

2023-04-20

Hervé Pagès (01:27:21) (in thread): > This has nothing to do with lack of--no-vignettesoption. We stopped using that option on the daily builds last year and we’ve never set it back, so I don’t see the correlation with the TIMEOUTs that we observe every now and then these days. > > Last time this has happened, the problem is resolved without the effort in my side > And it resolved without effort on our side either. > > Is it possible for you to check if it can happen in your local machine? > Have you checked if it happens on your local machine? As I said, this looks like an intermittent problem. > And you still haven’t provided any of the details that I asked earlier about the remote resource so I can’t help more.

Chengyang Ji (02:26:43): > @Chengyang Ji has joined the channel

Y-h. Taguchi (04:15:22) (in thread): > I do not know how to build binary package in windows. I can manage only ubuntu. At least, in windows,

Y-h. Taguchi (04:16:42) (in thread): > As for remote access,https://www.bioconductor.org/packages/devel/bioc/html/STRINGdb.htmltries to download external data set and took long - Attachment (Bioconductor): STRINGdb (development version) > The STRINGdb package provides a R interface to the STRING protein-protein interactions database (https://string-db.org).

Y-h. Taguchi (05:13:27) (in thread): > @Hervé PagèsI tried to perform “check” tool in Rstudio in windows. Since it does not have PDFlatex, it cannot complete the session, but till starting PDFlatex, it took only 8 minutes. Thus, it is ulikely to take 2440 secs. Is it what you requires?

Y-h. Taguchi (05:47:56) (in thread): > @Hervé PagèsHere is a result of devtools::check() in my windows notebook > > start <- Sys.time() > devtools::check() > end <- Sys.time() > > diff <- end - start > diff > Time difference of 8.681239 mins > > Does it make sense?

Hervé Pagès (11:00:00) (in thread): > Yes, that’s what we usually observe on the build machines too. Most of the time it only takes a few minutes forR CMD checkto complete. And sometimes,R CMD checkgets stuck in the “checking re-building of vignette outputs” step forever on our Linux server. That’s what I mean by “intermittent problem”. The fact that you did oneR CMD checkrun on Windows and got it to complete in 8 min does not help understanding this intermittent problem on Linux. You’d have to be on Linux and tryR CMD checkmultiple times. Do you think you could try this on our Docker image?

Y-h. Taguchi (11:28:25) (in thread): > @Hervé PagèsI did same on wsl and I got > > Time difference of 7.426313 mins > > Thus, 2400 sec is not the problem in my side. Do you agree？

Hervé Pagès (12:18:25) (in thread): > “wsl” is pretty vague. What version of Linux? What version of pandoc? (we have pandoc 2.7.3 on nebbiolo1, which is the version bundled with Ubuntu 22.04 LTS). Using our Docker container will give you the closest setup to what we use on our Linux builder (but you might need to install a few additional components that the Docker image is missing). Anyways, intermittent problems are tricky to troubleshoot because, by definition, they are hard to reproduce. It can take a lot of time to arrive to a clear understanding of the situation. Repeating that the problem is not on your side won’t really help towards reaching that goal.

Y-h. Taguchi (12:33:12) (in thread): > @Hervé PagèsOK. You do not think the testing on wsl makes sense. You trust only docker. I do not get used to use docker. This means that all bioconductor maintenors are required to get used to docker, is it your requirement? I did not assume such a requirement.

Hervé Pagès (12:50:11) (in thread): > All I’m saying is that if you want to troubleshoot a problem that we only see on one platform (Ubuntu 22.04 in this case), you need to be on a similar system. Our Docker image is one way to get access to such system. Testing on wsl is not meaningful without knowing what Linux versions you used and what version of pandoc. You need to understand that there are possibly hundreds of components involved in aR CMD checkrun of a package that has 200+ deps (direct and indirect). So the closest you can stick to the setup where that intermittent problem is observed, the more chances you have to be able to reproduce it. > We’re not requiring package maintainers to get used to docker. But we expect reasonable cooperation to help understand and troubleshoot problems affecting their package on Linux. Using the docker is one way to do that, but feel free to use something else as long as you use the same setup as on our Linux builder.

Y-h. Taguchi (16:55:13) (in thread): > @Hervé PagèsI understand what you mean but the system that requires the knowledge about Docker and the persomal test on Docker might not be sustainable. Now build on linux seems to be ok.http://bioconductor.org/checkResults/devel/bioc-LATEST/TDbasedUFEadv/

2023-04-21

Hervé Pagès (14:02:37): > @Kasper D. Hansen@Henrik BengtssonAbout thishttps://bioconductor.org/checkResults/3.17/bioc-LATEST/affxparser/palomino3-install.html(breaks many packages on Windows that depend onaffxparser). Is this because of a Windows-specific problem related to R 4.3 switching the default C++ compiler from C++14 to C++17? Maybe it’s a simple matter of addingC++14toSystemRequirements?

Kasper D. Hansen (14:13:59): > That’s a good hypothesis

Kasper D. Hansen (14:14:22): > I think we also need to change something in Makevars.win to switch C++ compiler

Henrik Bengtsson (14:36:06): > It’s tracked at <https://github.com/HenrikBengtsson/affxparser/issues/48>. It is reproducible there on GitHub Actions, meaning any fix can be tested there, including via PRs. I have no idea what the fix would be and, unfortunately, I won’t have time to look at that anytime soon. Hopefully, someone else can figure this out.

Kasper D. Hansen (14:40:15): > That’s nice Henrik. Herve is suggesting that we are being affected by R switching the “default” C++ version from 14 to 17 in R. That seems like a super reasonable “guess” (but doesn’t explain why this is Windows only). I’ll try to figure out how to tell R to use 14 instead of 17 on Windows and see if that fixes the problem on Github Action

2023-04-23

Kasper D. Hansen (16:01:35): > Ok, I have green badges on GA for this issue now. Trying to make a minimal fix before I commit to Bioc; the current code has a few things applied to it

Kasper D. Hansen (16:54:50): > Fix pushed to Bioc

Michael Milton (21:11:39): > @Michael Milton has joined the channel

2023-04-26

Lori Shepherd (15:40:31): > Thanks to all developers and community members for contributing to the project! Bioconductor 3.17 is now available! > Please see the full release announcement:https://bioconductor.org/news/bioc_3_17_release/

2023-05-03

Y-h. Taguchi (01:09:29): > Sorry for bothering you many times. TDbasedUFEadv has another error in the build.http://bioconductor.org/checkResults/devel/bioc-LATEST/TDbasedUFEadv/nebbiolo2-buildsrc.htmlThis seems to be because of system error since > 1. I have never committed anything since the last successful build > 2. This has happened only in development version, not in release version whereas those two are currently identical since I have never committed anything since the release of BioC 3.17 at 26th April > I am glad if you can fix this problem as soon as possible although you might be bothered with too many troubles in my TDbasedUFEadv……. > This seems to because of openSSL > “OpenSSL SSL_read: error:0A000126:SSL routines::unexpected eof while reading, errno 0”http://bioconductor.org/checkResults/devel/bioc-LATEST/TDbasedUFEadv/raw-results/nebbiolo2/buildsrc-out.txt

Andres Wokaty (09:50:25) (in thread): > I’ll take a look.

Kasper D. Hansen (10:40:44) (in thread): > In general, your package can break in devel without you doing anything, if a dependency changes. This is a feature. In practice, many dependency changes are the result of errors in the dependency and as a package author you just wait it out. But sometimes you need to take action. It can be hard to know which is which.

Y-h. Taguchi (22:06:05) (in thread): > @Kasper D. HansenThanks. I know that it can happen. But the above link said that the problem has occurred at > > Quitting from lines 93-106 (Enrichment.Rmd) > > https://code.bioconductor.org/browse/TDbasedUFEadv/blob/devel/vignettes/Enrichment.Rmdwhere I called only enrichrhttps://cran.r-project.org/web/packages/enrichR/index.htmlwhich is in CRAN not in Bioconductor. Thus it is strange that this errors occurred only in development version if it is caused by package dependency, since CRAN package unlikely has distinct dependency between Bioc release and Bioc Devel. > Thus, I believed that it was system problem.. - Attachment (code.bioconductor.org): Bioconductor Code: Browse > Browse the content of Bioconductor software packages. - Attachment (cran.r-project.org): enrichR: Provides an R Interface to ‘Enrichr’ > Provides an R interface to all ‘Enrichr’ databases. ‘Enrichr’ is a web-based tool for analysing gene sets and returns any enrichment of common annotated biological features. Quoting from their website ‘Enrichment analysis is a computational method for inferring knowledge about an input gene set by comparing it to annotated gene sets representing prior biological knowledge.’ See https://maayanlab.cloud/Enrichr/](https://maayanlab.cloud/Enrichr/)) for further details.

2023-05-04

Hervé Pagès (02:42:59) (in thread): > Hmm.. looks like I can reproduce this on nebbiolo2 by callingenrichr(genes, dbs)“enough” times. > Setup: > > library(enrichR) > genes <- c("ACTB", "ACTG1", "ALDOA", "ADAM6", "AEBP1", "ACTA2", "A2M") > setEnrichrSite("Enrichr") > dbs <- c("GO_Molecular_Function_2015", "GO_Cellular_Component_2015", "GO_Biological_Process_2015") > options(enrichR.quiet=TRUE) > > Then: > > for (i in 1:50) { cat(i, " "); enrichr(genes, dbs) } > # 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 Error in curl::curl_fetch_memory(url, handle = handle) : > # OpenSSL SSL_read: error:0A000126:SSL routines::unexpected eof while reading, errno 0 > > I tried the above several times and sometimes theforloop would complete without any problem, sometimes it would error like above, and sometimes it would get stuck like here: > > for (i in 1:50) { cat(i, " "); enrichr(genes, dbs) } > # 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 ^C > # Error in curl::curl_fetch_memory(url, handle = handle) : > # Operation was aborted by an application callback > > I interrupted after a few minutes with CTRL+C, which triggered the error.

Y-h. Taguchi (03:05:15) (in thread): > @Hervé PagèsThanks a lot. Under which circumstances did you try? On Docker? I have never tried multiple times thus I might have not recognized this problem at all, because I might have occasionally succeeded.

Martin Morgan (10:39:19): > https://bioconductor.org/checkResults/release/bioc-LATEST/BiocParallel/merida1-checksrc.htmlshows Rmpi not available on merida1 (also lconway for devel), preventing propagation of the package binaries on macOS. Can Rmpi be installed on these systems, or is there an alternative way (moving Rmpi toEnhances:?) to avoid Rmpi as a Suggests: package (it’s used in unit tests, the unit tests fail gracefully if it is not installed, but if functionality is not being tested it doesn’t seem like a good idea to continue supporting Rmpi…)?

Hervé Pagès (11:27:55) (in thread): > > On Docker? > on nebbiolo2 > > I have never tried multiple times > well, that the thing withintermittenterrors, you have to try multiple times if you want to have a chance to see them

Hervé Pagès (11:30:46) (in thread): > Please moveRmpitoEnhances:. CRAN doesn’t supportRmpion Mac and, even though we’ve tried to work around this in the past, it turned out to cause too many problems on our Mac builders, so we gave up recently. Sorry for that.

Kasper D. Hansen (12:35:54) (in thread): > Yes, the macOS defautl compilers etc. has essentially given up on Rmpi. There are beta installations for using it from Simon, but you need to recompile stuff, so highly non-trivial

Hervé Pagès (14:05:47) (in thread): > A much simpler way to reproduce this: > Setup: > > library(httr) > url <- "[https://maayanlab.cloud/Enrichr/enrich](https://maayanlab.cloud/Enrichr/enrich)" > genes <- c("ACTB", "ACTG1", "ALDOA", "ADAM6", "AEBP1", "ACTA2", "A2M") > body <- list(list=paste(genes, collapse="\n")) > > Then: > > for (i in 1:50) { cat(i, " "); httr::POST(url=url, body=body) } > # 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 Error in curl::curl_fetch_memory(url, handle = handle) : > # OpenSSL SSL_read: error:0A000126:SSL routines::unexpected eof while reading, errno 0 > > The above loop can either complete successfully, or return an error after a few iterations, or get stuck (timeout) after a few iterations. > I observe this on nebbiolo1 and nebbiolo2. Both machines are running Ubuntu 22.04. Some googling around suggests that this issue is specific to Ubuntu 22.04 and it happens with certain websites only (https://maayanlab.cloud/in this case).

Y-h. Taguchi (17:56:35) (in thread): > @Hervé PagèsThank you so much for your detailed investigation. Does this mean that I have nothing to do to resolve this problem other than stopping using EnrichR?

Hervé Pagès (19:18:59) (in thread): > I can think of various things you can do. 1) Let theEnrichRmaintainer know about this, maybe they will be able to come up with a workaround. 2) Report the problem to thehttrfolks, they might be aware of this issue and maybe are working on a fix. 3) Contact the people running the Enrichr service athttps://maayanlab.cloud/Enrichr/enrich. The problem only seems to happen with certain websites so maybe they can do something to address this on the server side. These 3 actions are not mutually exclusive.

Y-h. Taguchi (19:19:55) (in thread): > @Hervé PagèsThanks. I will do.

Y-h. Taguchi (20:35:13) (in thread): > I have posted an issue to Slack Channel of enrichrhttps://github.com/MaayanLab/enrichr_issues/issues/59as well as sent e-mails to enrichR and httr maintainer. Let’s see what’s going on. > ================= > Sorry for the bothering you. > I am a maintainer of Bioconductor package TDbasedUFEadvBioconductor - TDbasedUFEadvthat uses ******** that I believe that you maintain. > Recently, TDbasedUFEadv has had build problem and it turned out that it is because of ********. > For details seehttps://community-bioc.slack.com/archives/CEQ04GKEC/p1683090569379919In order to see the above you have to subscribe Bioconductor slack channel that you can freely join. > I am glad if you can solve this problem as soon as possible, since I used ******** in my package. > ======= > ******** = enrichR or httr - Attachment (Bioconductor): TDbasedUFEadv > This is an advanced version of TDbasedUFE, which is a comprehensive package to perform Tensor decomposition based unsupervised feature extraction. In contrast to TDbasedUFE which can perform simple the feature selection and the multiomics analyses, this package can perform more complicated and advanced features, but they are not so popularly required. Only users who require more specific features can make use of its functionality. - Attachment: Attachment > Sorry for bothering you many times. TDbasedUFEadv has another error in the build. > http://bioconductor.org/checkResults/devel/bioc-LATEST/TDbasedUFEadv/nebbiolo2-buildsrc.html > This seems to be because of system error since > 1. I have never committed anything since the last successful build > 2. This has happened only in development version, not in release version whereas those two are currently identical since I have never committed anything since the release of BioC 3.17 at 26th April > I am glad if you can fix this problem as soon as possible although you might be bothered with too many troubles in my TDbasedUFEadv……. > This seems to because of openSSL > “OpenSSL SSL_read: error:0A000126:SSL routines::unexpected eof while reading, errno 0” > http://bioconductor.org/checkResults/devel/bioc-LATEST/TDbasedUFEadv/raw-results/nebbiolo2/buildsrc-out.txt - Attachment: #59 [BUG] > Sorry for the bothering you.
> I am a maintainer of Bioconductor package TDbasedUFEadv
> Bioconductor - TDbasedUFEadv
> that uses Enrichr that I believe that you maintain.
> Recently, TDbasedUFEadv has had build problem and it turned out that it is because of Enrichr.
> For details see
> https://community-bioc.slack.com/archives/CEQ04GKEC/p1683090569379919
> In order to see the above you have to subscribe Bioconductor slack channel that you can freely join.
> I am glad if you can solve this problem as soon as possible, since I used Enrichr in my package.

2023-05-06

Y-h. Taguchi (05:56:08) (in thread): > I have gotten the negative responses from them. They are not willing to address this issue, since no others have problems at all. We need to find some others who have similar problems….

2023-05-08

Michael Milton (00:00:34): > Who specifically has their GitHub SSH keys added to the BioC git server? It’s not entirely explainedhere. Is it thecre, or theautas well? I currently getFATAL: W any packages/CuratedAtlasQueryR stemangiola DENIED by fallthruif I try to push.

Hervé Pagès (13:03:47) (in thread): > You’d probably want to ask this in the#bioc_gitchannel.

2023-05-09

Steffen Neumann (02:40:58): > Hi, the xcms Vignettes onhttp://bioconductor.org/packages/3.18/bioc/html/xcms.htmlare missing.

Steffen Neumann (02:41:59) (in thread): > Is that because Linux has a red dot onhttp://bioconductor.org/checkResults/devel/bioc-LATEST/xcms/which says dependency MSnbase is missing ?@Johannes Rainerand I are wondering …

Steffen Neumann (02:47:28) (in thread): > Ah, and that goes deep down the rabbit hole via affy and rsqlite and whatnot. Grabbing popcorn and waiting for things to resolve. Unless I can help somewhere

Andres Wokaty (09:10:11) (in thread): > Yes, it’s not getting propagated because of the missing dependency.

2023-05-17

Sarvesh Nikumbh (07:29:27): > Hi, my package seqArchR is failing on devel/Linux (nebbiolo2) due to missing Python modules (scikit-learn).https://bioconductor.org/checkResults/3.18/bioc-LATEST/seqArchR/nebbiolo2-checksrc.html@Andres WokatyCould you please look into this? > Thanks in advance!

2023-05-18

Oluwafemi Oyedele (05:54:04): > @Oluwafemi Oyedele has joined the channel

2023-05-20

Franck RICHARD (06:15:08): > @Franck RICHARD has joined the channel

2023-05-30

Kasper D. Hansen (11:21:18): > What is happening with the release builds on macOS arm64?

Kasper D. Hansen (11:21:51): > I see onhttps://bioconductor.org/packages/release/bioc/html/BiocGenerics.htmlthat the version picked up by an installation on release is 0.45.3 and not 0.46 which is the actual relase - Attachment (Bioconductor): BiocGenerics > The package defines many S4 generic functions used in Bioconductor.

Kasper D. Hansen (11:22:34): > It is also behind on a number of other packages; here is from a fresh installation

Kasper D. Hansen (11:22:43): > > There are binary versions available but the source versions are later: > binary source needs_compilation > Rhtslib 2.1.1 2.2.0 TRUE > BiocGenerics 0.45.3 0.46.0 FALSE > DelayedArray 0.26.2 0.26.3 TRUE > Rsamtools 2.15.2 2.16.0 TRUE > BiocParallel 1.34.1 1.34.2 TRUE >

Kasper D. Hansen (11:23:03): > Also note thatRsamtoolis not the actual stable release

Lori Shepherd (11:27:51): > https://support.bioconductor.org/p/9152542/

Lori Shepherd (11:28:34): > It is actually failing – I track BiocGenerics back to RProtoBufLib — which maybe has a stray lock file maybe from a previous build failure –@Andres Wokatyshould be looking into it

Kasper D. Hansen (11:32:05): > Is this because offlowClustfailing as the support message says?

Kasper D. Hansen (11:32:35): > BiocGenericsonly Suggests flowclust because there is an upstream link to a man page

Kasper D. Hansen (11:32:52): > > \item \link[flowClust]{density,flowClust-method} in the \pkg{flowClust} > package for an example of a specific \code{density} method (defined > for \link[flowClust]{flowClust} objects). >

Kasper D. Hansen (11:33:48): > This seems a bit crazy to me, that we have an absolute core package failing in release because of a missing man page. I mean, I understand the reasons why, but this is too brittle

Kasper D. Hansen (11:34:15): > Not sure I have a good suggestion on how to fix this in general, but in this specific case we should remove that man page reference

Lori Shepherd (11:34:50): > I’m not sure where you pulled that — but if I follow it down in 3.17 > BiocGenerics > flowClust > flowCore > cytolib/RProtoBuff — RProtoBuff because of a lock file — nothing about a man page

Kasper D. Hansen (11:36:37): > BiocGenericsonly SuggestsFlowClustbecause it references a flowClust man page

Kasper D. Hansen (11:36:53): > I understand that the reason why flowClust itself fails may be complicated

Kasper D. Hansen (11:37:56): > But I don’t think we want something like BiocGeneric to depend on a whole stack of flow cytometry packages only for a man page link - I say this as someone who appreciate links to man pages

Kasper D. Hansen (11:38:29): > Rsamtoolsfail because ofRhtslibnot installing because of a 00LOCK file. That should be easy

Kasper D. Hansen (11:40:57) (in thread): > As far as I can see when I searchBiocGenerics. It also makes sense -BiocGenericsshould only have generics so little-to-no depend

Andres Wokaty (12:47:38): > I removed all the locks that were there. Unfortunately since this machine is slow, the updated binaries won’t be available until next week.

Alan O’C (13:42:44) (in thread): > It’s also just pointing to an arbitrary implementation of an s4 density method. Maybe just point to a way to search bioc for methods that attach to the generic?

Kasper D. Hansen (15:56:46) (in thread): > That would be better and more robust for this specific question.

2023-06-02

kent riemondy (14:24:25): > @kent riemondy has joined the channel

2023-06-07

Kayla Jackson (13:51:41): > @Kayla Jackson has joined the channel

Alyssa Obermayer (18:28:17): > @Alyssa Obermayer has joined the channel

2023-06-08

Hervé Pagès (14:29:18): > We’re adding Linux ARM64 builds to our daily builds:https://bioconductor.org/checkResults/3.18/bioc-LATEST/long-report.htmlThis is still experimental! > Thanks to Martin Grigorov and Yikun Jiang for running the Linux ARM64 builds and for helping us improve support for this new platform. > Note that Martin wrote a blog post herehttps://martin-grigorov.medium.com/emulated-building-and-testing-of-bioconductor-projects-for-linux-arm64-501c1f854dc1about how to debug Linux ARM64 related issues on a x86_64 host. That link is displayed on the pages for kunpeng2 results in the dotted box that contains notes to the developers. See for examplehttps://bioconductor.org/checkResults/3.18/bioc-LATEST/rsbml/kunpeng2-install.html

Marcel Ramos Pérez (14:32:20) (in thread): > Can they repost in the Bioc blog?

Hervé Pagès (14:33:11) (in thread): > up to them I guess

Martin Grigorov (15:06:34) (in thread): > Sure! How to do it?

Marcel Ramos Pérez (15:19:43) (in thread): > Thank you! Here are the instructions :https://bioconductor.github.io/biocblog/contributing.html

2023-06-09

Martin Grigorov (04:48:46) (in thread): > Done:https://github.com/Bioconductor/biocblog/pull/43

2023-06-12

Davide Risso (08:31:29): > Hi everyone, we found a weird issue in trying to install the NewWave package in Bioc 3.17. Specifically, > > BiocManager::install("NewWave") > > returns this error message > > Warning message: > package 'NewWave' is not available for Bioconductor version '3.17' > > A version of this package for your version of R might be available elsewhere, > see the ideas at[https://cran.r-project.org/doc/manuals/r-patched/R-admin.html#Installing-packages](https://cran.r-project.org/doc/manuals/r-patched/R-admin.html#Installing-packages) > > but the package exists in the build system:http://bioconductor.org/checkResults/release/bioc-LATEST/NewWave/The following workaround works, confirming that the binary is available > > install.packages("[https://bioconductor.org/packages/release/bioc/bin/macosx/big-sur-x86_64/contrib/4.3/NewWave_1.10.0.tgz](https://bioconductor.org/packages/release/bioc/bin/macosx/big-sur-x86_64/contrib/4.3/NewWave_1.10.0.tgz)", repos = NULL) > > This happens both on Intel Mac and Windows (didn’t try with other OSs)

Martin Grigorov (08:33:23) (in thread): > try with, type = "source"

Martin Grigorov (08:33:36) (in thread): > > BiocManager::install("NewWave", type = "source") >

Martin Grigorov (08:34:30) (in thread): > note the red circle for Linux athttp://bioconductor.org/checkResults/3.17/bioc-LATEST/NewWave/

Martin Grigorov (08:34:40) (in thread): > hover over it to see the reason/error

Davide Risso (08:35:46) (in thread): > I already tried thetype="source"with no success (same message) and I also noticed the broken linux (but it’s available for Mac and Win)

Martin Grigorov (08:38:40) (in thread): > sorry, I just noticed that you tried on Intel Mac and Windows. Linux should not be related then

Mike Smith (08:48:15) (in thread): > Hi@Davide Risso. For other Linux users at CSAMA I got them to install directly from the git server: > > remotes::install_git("[https://git.bioconductor.org/packages/SharedObject](https://git.bioconductor.org/packages/SharedObject)", ref = "RELEASE_3_17") > remotes::install_git("[https://git.bioconductor.org/packages/NewWave](https://git.bioconductor.org/packages/NewWave)", ref = "RELEASE_3_17") > > The SharedObject package has a build error on Linux, and that’s stopping the propagation of NewWave. However the error is just a unit test failure that isn’t present on the other platforms, so I decided it was fairly safe to let users install SharedObject manually. > > I hoped to try and document all the weird workarounds I figured out for the various package/OS combination at CSAMA, but didn’t have time over the weekend.

Mike Smith (08:52:55) (in thread): > I haven’t figured out why the lack of an available source package prevents installation of the binary without specifically asking for it. Presumably something inBiocManager::install()stops before it gets to looking for the binary.

Davide Risso (09:00:28) (in thread): > Thanks Mike! Yeah, to me this is the most plausible explanation: BiocManager thinks the binary doesn’t exist and doesn’t even look for it

Mike Smith (09:12:16) (in thread): > It feels to me like there are been a few more problems in getting everything distributed following the release of BioC 3.17 than in previous versions. However, it might just be a bias in when something like CSAMA has fallen relative to the release cycle, so I’m noticing it more acutely. The timeout of Basiilisk on Mac (https://www.bioconductor.org/packages/release/bioc/html/basilisk.html) is pretty annoying as it’s preventing the distribution of quite a few Mac binaries, and forcing the installation of xcode/gfortran etc. I’d rather not have to support that for the students if it was possible to avoid. - Attachment (Bioconductor): basilisk > Installs a self-contained conda instance that is managed by the R/Bioconductor installation machinery. This aims to provide a consistent Python environment that can be used reliably by Bioconductor packages. Functions are also provided to enable smooth interoperability of multiple Python environments in a single R session.

Vince Carey (09:36:24) (in thread): > I am ok with dropping basilisk for student installations.

Mike Smith (09:49:17) (in thread): > Unfortunately basilisk is a (indirect) dependency for quite a few packages, so its not easy to just waive it. Specifically in this case I think it’s OSCA -> scater -> densvis -> basilisk but there may well be other routes too. The list of packages for CSAMA is pretty long and I didn’t audit anything. We just compile a complete list from that document in the repository. > > The nice thing is that it’s actually easy to install basilisk from source, so I don’t think it’s been a problem to install it for anyone. However the lack of a basilisk Mac binary prevents the propagation of the binaries for the dependent packages, even though they build fine. In this case it means the students need all the compilers etc to install densvis, even though the build system has some point produced binaries (but not made them available).Caveat: This is my reading of the situation, it might be wrong.Anyway, I think we got everyone who contacted us a working installation, so hopefully it isn’t an issue during the course, but it did require a few clunky fixes and hardcoded install lines for specific package/platform combos in the installation script.

Davide Risso (09:58:11) (in thread): > I can confirm that in my Mac Ventura the most problematic package to install was densvis, which took some time to figure out the right Fortran version.

Vince Carey (10:34:54) (in thread): > Thanks for your efforts Mike. I hope folks at the BBS can help make the binaries accessible.

Hervé Pagès (12:10:52) (in thread): > How? If I understand correctly Mike’s analysis of the situation, the cause for many missing Mac binaries is abasilisktimeout on Mac. This would need to be addressed inbasiliskitself.

Vince Carey (12:16:43) (in thread): > @Hervé Pagèsyes, but in devel branch, where tests were moved to longtests, the situation does not seem to have resolved. There is an M1 mac binary accessible on the landing pagehttps://bioconductor.org/packages/3.17/bioc/html/basilisk.htmlbut not one for intel mac. I wondered if perhaps there is a usable binary available or if one could be made available as a one-off. I don’t have a mac nearby. - Attachment (Bioconductor): basilisk > Installs a self-contained conda instance that is managed by the R/Bioconductor installation machinery. This aims to provide a consistent Python environment that can be used reliably by Bioconductor packages. Functions are also provided to enable smooth interoperability of multiple Python environments in a single R session.

Vince Carey (12:17:48) (in thread): > I am not insisting on this, just wondering if there is a way past this obstacle.

Hervé Pagès (12:29:38) (in thread): > Moving somebasilisk’s tests to the long tests only reducedR CMD checktime from 1560.9 sec (release) to 1500.3 sec (devel) on Linux, so it’s a good start but more will need to be done to solve the timeout problems. Also these fixes will need to be applied to the RELEASE_3_17 branch.

Hervé Pagès (12:56:00) (in thread): > @Martin GrigorovThanks for doing that. Should we replace the link to your personal blog post withhttps://bioconductor.github.io/biocblog/posts/2023-06-09-debug-linux-arm64-on-docker/on the daily report? - Attachment (bioconductor.github.io): Bioconductor community blog - Emulated build and test of Bioconductor packages for Linux ARM64 > Build and test for Linux ARM64 with Docker on x86_64 host

Martin Grigorov (12:56:45) (in thread): > I already opened a PR for this

Hervé Pagès (12:57:51) (in thread): > Ah, I missed that. Seeing it now. Will merge. Thanks!

Davide Risso (15:23:48) (in thread): > In the case of NewWave, my understanding is that the issue is with SharedObject and not basilisk

Mike Smith (15:27:07) (in thread): > I wonder if there is an argument that the build system could make the binary for denvis (in this case) available, since that works, even though the process to produce the basilisk binary was not successful. > > As demonstrated by this example,BiocManager::install()will happily try to install the source version of basilisk by default if the binary isn’t found. What might be the issue if a binary of densvis was then installed? Could there be any repercussions to that? I guess maybe some mixed compiler linking stuff, but that is always a risk when we distribute binaries and we already implement a lot of strategies to mitigate that e.g static linking. In this case I think it’s moot as basilisk doesn’t require any compilation to install from source.

Mike Smith (15:29:04) (in thread): > I think the issue with SharedObject is a separate problem, and it comes down to whether it was the Linux builder or Mac which didn’t succeed, and then how BiocManager copes with the missing result e.g no binary Vs no source tarball.

Hervé Pagès (15:56:05) (in thread): > > I wonder if there is an argument that the build system could make the binary for denvis (in this case) available, since that works, even though the process to produce the basilisk binary was not successful. > I think that would be a reasonable thing to do. The current criteria for propagating a binary package is that all deps must be available as binaries (except if the deps are data-ann or data-exp packages). But we could refine this logic and still allow propagation when the dep is available as a source tarballandhasNeedsCompilationset toNo, which is the case forbasiliskand many other software packages: > > > avpkgs <- available.packages(contrib.url("[https://bioconductor.org/packages/3.18/bioc](https://bioconductor.org/packages/3.18/bioc)")) > > avpkgs["basilisk" , "NeedsCompilation"] > [1] "no" > > We’ll work on this.

Hervé Pagès (15:57:31) (in thread): > I should have said “formostsoftware packages”: > > > table(avpkgs[ , "NeedsCompilation"]) > > no yes > 1470 452 >

Alan O’C (17:34:54) (in thread): > I can move the basilisk packages to Suggests in scater if that helps?

2023-06-13

Hervé Pagès (01:32:44) (in thread): > So propagation criteria for binary packages has been relaxed. We’ll see how this helps more binary packages to propagate later this morning.@Alan O’Cprobably not necessary

Mike Smith (03:44:37) (in thread): > This is awesome, thanks so much@Hervé Pagèsfor being so responsive. I think this will ease some installation headaches for quite a few users.

Alan O’C (04:11:57) (in thread): > Cool cheers, I may end up doing it as prophylaxis anyways as I want to reduce the amount of time I spend fixing build failures. As always feel free to shout if I can make things easier on yous as well

Hervé Pagès (14:25:07) (in thread): > @Davide RissoIn the case ofNewWave, it looks like a bug ininstall.packages(..., type="both")to me (install.packages()is the workhorse behindBiocManager::install()). > Here is what I get on a Mac: > > By defaultinstall.packages()usestype="both"and doesn’t see theNewWavebinary: > > > getOption("pkgType") > [1] "both" > > install.packages("NewWave", repo="[https://bioconductor.org/packages/3.17/bioc](https://bioconductor.org/packages/3.17/bioc)") > Warning message: > package 'NewWave' is not available for this version of R > > A version of this package for your version of R might be available elsewhere, > see the ideas at[https://cran.r-project.org/doc/manuals/r-patched/R-admin.html#Installing-packages](https://cran.r-project.org/doc/manuals/r-patched/R-admin.html#Installing-packages) > > However if we specifytype="mac.binary", theninstall.packages()sees the binary: > > > install.packages("NewWave", repo="[https://bioconductor.org/packages/3.17/bioc](https://bioconductor.org/packages/3.17/bioc)", type="mac.binary") > trying URL '[https://bioconductor.org/packages/3.17/bioc/bin/macosx/contrib/4.3/NewWave_1.10.0.tgz](https://bioconductor.org/packages/3.17/bioc/bin/macosx/contrib/4.3/NewWave_1.10.0.tgz)' > > > Content type 'application/x-gzip' length 413389 bytes (403 KB) > ================================================== > downloaded 403 KB > > > The downloaded binary packages are in > /var/folders/v1/y6dg5h4n163dzmrfl6t_r5480000gp/T//RtmpV622cc/downloaded_packages > > Note that the bug is actually inavailable.packages()whichinstall.packages()uses internally to obtain the list of packages that are available: > > > avpkgs <- available.packages(repos="[https://bioconductor.org/packages/3.17/bioc](https://bioconductor.org/packages/3.17/bioc)", type="both") > > c("BiocGenerics", "NewWave") %in% avpkgs[ , "Package"] > [1] TRUE FALSE > > :unamused:

Hervé Pagès (14:35:52) (in thread): > To provide some historical context I think that when the R folks introducedtype="both"about 10+ years ago the primary use case was to handle the situation where the binary is not avalaible but the source is, soinstall.packages()can fall back on the source (granted it doesn’t need compilation). So they focused on that. But the opposite situation where the binary is available and the source is not should work too, even if it’s a much less common situation, so rare indeed that this bug has been unnoticed for 10+ years!

Hervé Pagès (15:06:22) (in thread): > Reported:https://bugs.r-project.org/show_bug.cgi?id=18554

Mike Smith (15:22:17) (in thread): > Great detective work! Thanks for doing the deep digging to confirm my suspicions about what was happening.

Davide Risso (17:14:58) (in thread): > Thank you Hervé! I would have never guessed this!

Hervé Pagès (23:56:04) (in thread): > Issue got closed (https://bugs.r-project.org/show_bug.cgi?id=18554) and won’t be fixed.:face_with_rolling_eyes:

2023-06-14

Alan O’C (02:52:28) (in thread): > Not that surprising given that R core is GPL

Hervé Pagès (11:29:33) (in thread): > but packages on CRAN are not necessarily GPL

Hervé Pagès (13:28:22) (in thread): > allbasiliskbinary rev deps made it to the 3.17 repo today:https://bioconductor.org/checkResults/3.17/bioc-LATEST/basilisk/:tada:(see little green LEDs on the right)

Hervé Pagès (14:05:00): > @Martin Grigorov@Yikun JiangIt looks like some TeX/LaTeX components are missing on kunpeng2, causingR CMD buildto fail for the following packages: BEAT, BiocHail, chromstaR, diffuStats, eegc, gcatest, GUIDEseq, hierinf, lfa, MultiMed, piano, podkat, qckitfastq, RGMQL. Do you think you can take a look at this? FWIW here’s the list of TeX/LaTex-related.debpackages that we install on our Ubuntu builders to support the creation of all Bioconductor vignettes. Thanks!

Hervé Pagès (14:05:43): > https://github.com/Bioconductor/BBS/blob/devel/Ubuntu-files/22.04/apt_vignettes_reference_manuals.txt

Martin Grigorov (15:54:58): > Hi@Hervé Pagès! Yes, we are aware of this problem. I have mentioned those packages athttps://github.com/Bioconductor/BBS/pull/293#discussion_r1217939963. We are working on adding the missing packages to openEuler OS!

Hervé Pagès (17:12:58): > The comment at the above link says: > > Some OS dependencies are not available yet in openEuler and because of this they are installed from source. > but in this case it seems that the OS dependencies are not installed at all. Do you think they can be installed from source? I’d understand if you prefer to wait that they become available as openEuler RPMs though.

Alan O’C (18:13:02) (in thread): > Yeah, I don’t mean it’s correct, but that it’s unsurprising

Hervé Pagès (20:01:05) (in thread): > > > cran_url <- "[https://cran.r-project.org](https://cran.r-project.org)" > > cran_pkgs <- available.packages(contrib.url(cran_url, type="source")) > > is_gpl <- grepl("\\<gpl\\>", cran_pkgs[ , "License"], ignore.case=TRUE) > > sum(is_gpl) / length(is_gpl) > [1] 0.7061843 > > so only 70% unsurprising:wink:… And this is assuming that this was the motivation or inspiration for the currenttype="both"behavior, but I would be 100% surprised if that was the case:face_with_spiral_eyes:

2023-06-16

Kevin Rue-Albrecht (15:40:32): > something wrong withBiocCheck()or more likely something it queries today? I’m getting a lot of > > * Checking for deprecated package usage... > Error in `[.data.frame`(pkg_status, , "PackageStatus") : > undefined columns selected >

Kevin Rue-Albrecht (15:42:35): > seems to be coming from here:https://github.com/Bioconductor/BiocCheck/blob/da640118615b4fdb6f9a5357d4f34c6dcd2a066b/R/util.R#L233

Kevin Rue-Albrecht (15:54:10): > nevermind, I just found that the package was bugfixed 4 hours ago:slightly_smiling_face:https://github.com/Bioconductor/BiocCheck/commit/eebccc701c04d457ea96c99008bfa7eb3716e5a1

2023-06-18

Yikun Jiang (23:18:53) (in thread): > Sorry for late reply, I also take a deep look on these package failure

Yikun Jiang (23:18:57) (in thread): > > BEAT (fixed) > dnf install texlive-esint-type1 > > # TO be installed > eegc, diffuStats > dnf install texlive-doublestroke > > gcatest, lfa > dnf install texlive-was > > hierinf, > dnf install texlive-eepic > > MultiMed > dnf install texlive-xypic > > piano > dnf install texlive-babel-swedish > > podkat > dnf install texlive-cm-mf-extra-bold > > qckitfastq > dnf install texlive-makecell >

Yikun Jiang (23:20:09) (in thread): > {chromstar} > LaTeX Error: Unknown graphics extension: .2-1. > > GUIDEseq > LaTeX errors: > ! Missing \endcsname inserted. > > Above 2 packages, I haven’t found the sepecific rpm to install…

Yikun Jiang (23:20:37) (in thread): > {biochail} > Low version spark (NOT SUPPORTED also in win/Mac) > > it will not be supported just like win/mac, because spark only support linux aarch64 after 3.3.0 version

Yikun Jiang (23:20:41) (in thread): > cc@Martin Grigorov

2023-06-19

Martin Grigorov (02:34:41) (in thread): > We have already installed several libraries from source (e.g. jags, libsbml, udunits2, gsl, openbabel, pandoc, xpdfviewer). Those seem to be used by many Bioc packages. > We will install more once we identify them!

2023-06-20

Hervé Pagès (15:16:02) (in thread): > @Yikun JiangThanks for looking at this!

2023-06-21

Nicholas Cooley (14:19:08): > Hi, we’ve been having a build error on palomino for a bit that I’m not sure what to do with:http://bioconductor.org/checkResults/devel/bioc-LATEST/SynExtend/palomino4-buildsrc.html > > * creating vignettes ... ERROR > --- re-building 'UsingSynExtend.Rmd' using rmarkdown > !!! Error: Cannot call Ghostscript: `mgs' (No such file or directory/The system cannot find the file specified)! > ! LaTeX Error: Command \@raggedtwoe@everyselectfont undefined. > > Error: processing vignette 'UsingSynExtend.Rmd' failed with diagnostics: > LaTeX failed to compile F:/biocbuild/bbs-3.18-bioc/tmpdir/RtmpEB39ux/Rbuild10bc61e77674/SynExtend/vignettes/UsingSynExtend.tex. See[https://yihui.org/tinytex/r/#debugging](https://yihui.org/tinytex/r/#debugging)for debugging tips. See UsingSynExtend.log for more info. > --- failed re-building 'UsingSynExtend.Rmd' >

Hervé Pagès (14:32:25) (in thread): > Looks like aknitr/tinytexissue to me. Easy workaround would be to replaceBiocStyle::pdf_documentwithBiocStyle::html_documentin the vignette.

Nicholas Cooley (14:58:26) (in thread): > ohp, i’ll try that, thanks!

Hervé Pagès (15:08:39) (in thread): > There are a few pending MikTeX updates that we need to apply on the Windows builders. Jen is going to take care of that before the end of the week. Maybe this will help, who knows.

2023-06-22

Steffen Neumann (09:28:04): > Hi, could someone remind me how to break circular dependencies on the build system ?

Steffen Neumann (09:29:29) (in thread): > XCMS won’t build because faahKO is not availablehttp://bioconductor.org/checkResults/devel/bioc-LATEST/xcms/nebbiolo2-buildsrc.htmlwhile faahKO won’t propagate because xcms is not available.http://bioconductor.org/checkResults/devel/data-experiment-LATEST/faahKO/

Andres Wokaty (11:23:35) (in thread): > I updated Miktex and tried building the package but got the same error.

Hervé Pagès (12:22:22) (in thread): > Thanks Jen. It was worth trying:disappointed:

2023-06-23

Lori Shepherd (07:04:26) (in thread): > @Andres Wokaty/@Hervé PagèsI thought this would resolve automatically but maybe not anymore with the new setup?

Andres Wokaty (10:45:48) (in thread): > It should be resolved automatically. I see faaKO depends on xcms, which suggests faahKO. I updated R on most of the machines Tuesday and the build system usually needs a few days to reinstall everything. Data Experiments ran yesterday, so I see it’s starting to turn green on the linux machines but the mac and windows machines will need a little more time to catch up.

Hervé Pagès (11:54:16) (in thread): > xcmsnow builds (on today’s report) but it won’t propagate becauseaffyis not available (xcmsdepends indirectly onaffyviaMSnbaseandvsn).

Lori Shepherd (13:02:04) (in thread): > affy is being worked on and should hopefully be fixed shortly

2023-06-29

Alan O’C (07:07:36): > scater is failing on devel because ofhttps://github.com/Bioconductor/MatrixGenerics/issues/31I don’t seem to be able to fix that warning on my end. With current MatrixGenerics and DelayedMatrixStats, I get the same behaviour whether or not I specify useNames: > > > rowVars(DelayedArray(matrix(1))) > [1] NA > Warning message: > useNames = NA is deprecated. Instead, specify either useNames = TRUE or useNames = TRUE. > > rowVars(DelayedArray(matrix(1)), useNames=TRUE) > [1] NA > Warning message: > useNames = NA is deprecated. Instead, specify either useNames = TRUE or useNames = TRUE. > > Is there an upstream fix? - Attachment: #31 ‘useNames = TRUE’ is the new default in matrixStats v1.0.0 > matrixStats v1.0.0 was just published on CRAN (HenrikBengtsson/matrixStats#232). The main change is that the old default useNames = NA is now deprecated and the new default useNames = TRUE. > > I wonder if we should patch the current release version to use useNames = FALSE because that avoids deprecation warnings and won’t change the behavior all of a sudden. And then with the next Bioconductor release change the default to useNames = TRUE

Hervé Pagès (14:21:01) (in thread): > This is due to a recent non-backward compatible change inmatrixStats. We’ve addressed the issue in release (BioC 3.17) but not in devel yet. I submitted a PR forMatrixGenericsabout 3 weeks ago. Still waiting for a green light from Pete. There might be additional fixes that need to be applied tosparseMatrixStats,SparseArray,DelayedArrayandDelayedMatrixStats, and probably in a few other places, but it will be easier to identify them after theMatrixGenerics’s PR is merged.@Peter Hickey?

Peter Hickey (18:24:58) (in thread): > sorry, I started adaptingDelayedMatrixStatstohttps://github.com/Bioconductor/MatrixGenerics/pull/33/but hit an issue and then I’ve been caught up with other things at work - Attachment: #33 Apply matrixStats 1.0.0 breaking change for ‘useNames’ default [PR against devel] > All generic functions and methods defined in MatrixGenerics now set the useNames argument to TRUE instead of NA by default, for consistency with matrixStats 1.0.0. See #31 for a discussion of this change.

Peter Hickey (18:25:22) (in thread): > it’ll be next week before i have capacity to deal return to this

2023-06-30

Henrik Bengtsson (02:42:21) (in thread): > > There might be additional fixes that need to be applied tosparseMatrixStats,SparseArray,DelayedArrayandDelayedMatrixStats, and probably in a few other places, but it will be easier to identify them after theMatrixGenerics’s PR is merged. > I track broken packages inhttps://github.com/HenrikBengtsson/matrixStats/issues/227 - Attachment: #227 Packages breaking, because useNames=TRUE is the new default > With the new useNames = TRUE as the default, there are 1 broken Bioconductor release packages and 6 broken Bioconductor devel packages (2023-06-28). All CRAN packages are ok. > > CRAN > > ☑︎ Canek 0.2.1 (2022-05-10), 0.2.2 (2023-05-31) > ☑︎ Issue/PR: MartinLoza/Canek#12 (2023-03-21) > • Error: https://github.com/HenrikBengtsson/matrixStats/blob/58a8b54f571cb4fbae65b1bc4d738e2a2dd8e8b1/revdep/problems.md#canek|https://github.com/HenrikBengtsson/matrixStats/blob/58a8b54f571cb4fbae65b1bc4d738e2a2dd8e8b1/revdep/problems.md#canek > > Last updated: 2023-06-01 > > Bioconductor > > MatrixGenerics >
> • Issues: Bioconductor/MatrixGenerics#31 > ☑︎ Fixed in Bioc release (MatrixGenerics 1.12.1; 2023-06-10) > ☐ Fixed in Bioc devel > sparseMatrixStats >
> • Notes: Already prepared for the move, cf. https://github.com/const-ae/sparseMatrixStats/tree/change_useNames_default > • Remaining issues: const-ae/sparseMatrixStats#26 > ☑︎ Fixed in Bioc release > ☐ Fixed in Bioc devel > DelayedMatrixStats >
> • Issue: PeteHaitch/DelayedMatrixStats#91 > ☑︎ Fixed in Bioc release > ☐ Fixed in Bioc devel > BASiCS >
> ☑︎ Fixed in Bioc release > ☑︎ Fixed in Bioc devel > • GitHub: https://github.com/catavallejos/BASiCS > • Notes: Imports only from matrixStats > • Error: https://github.com/HenrikBengtsson/matrixStats/blob/58a8b54f571cb4fbae65b1bc4d738e2a2dd8e8b1/revdep/problems.md#basics|https://github.com/HenrikBengtsson/matrixStats/blob/58a8b54f571cb4fbae65b1bc4d738e2a2dd8e8b1/revdep/problems.md#basics > CATALYST >
> ☑︎ Fixed in Bioc release (Check error is gone without package being updated; looks like there was an indirect dependency on MatrixStats, which has been updated since. /2023-06-14) > ☐ Fixed in Bioc devel > • GitHub: https://github.com/HelenaLC/CATALYST > • Notes: Imports only from matrixStats > • Error: https://github.com/HenrikBengtsson/matrixStats/blob/58a8b54f571cb4fbae65b1bc4d738e2a2dd8e8b1/revdep/problems.md#catalyst|https://github.com/HenrikBengtsson/matrixStats/blob/58a8b54f571cb4fbae65b1bc4d738e2a2dd8e8b1/revdep/problems.md#catalyst > glmGamPoi >
> ☑︎ Fixed in Bioc release > ☐ Fixed in Bioc devel > • GitHub: https://github.com/const-ae/glmGamPoi > • Notes: It uses functions from both matrixStats and DelayedMatrixStats > • Error: https://github.com/HenrikBengtsson/matrixStats/blob/58a8b54f571cb4fbae65b1bc4d738e2a2dd8e8b1/revdep/problems.md#glmgampoi|https://github.com/HenrikBengtsson/matrixStats/blob/58a8b54f571cb4fbae65b1bc4d738e2a2dd8e8b1/revdep/problems.md#glmgampoi > metagenomeSeq >
> ☐ Fixed in Bioc release > ☐ Fixed in Bioc devel > • GitHub: https://github.com/HCBravoLab/metagenomeSeq > • Notes: Imports only from matrixStats > • Error: https://github.com/HenrikBengtsson/matrixStats/blob/58a8b54f571cb4fbae65b1bc4d738e2a2dd8e8b1/revdep/problems.md#metagenomeseq|https://github.com/HenrikBengtsson/matrixStats/blob/58a8b54f571cb4fbae65b1bc4d738e2a2dd8e8b1/revdep/problems.md#metagenomeseq > scp >
> ☑︎ Fixed in Bioc release > ☑︎ Fixed in Bioc devel > • GitHub: https://github.com/UCLouvain-CBIO/scp > • Notes: Imports only from matrixStats > • Error: https://github.com/HenrikBengtsson/matrixStats/blob/58a8b54f571cb4fbae65b1bc4d738e2a2dd8e8b1/revdep/problems.md#scp|https://github.com/HenrikBengtsson/matrixStats/blob/58a8b54f571cb4fbae65b1bc4d738e2a2dd8e8b1/revdep/problems.md#scp > scPCA >
> ☑︎ Fixed in Bioc release (Check error is gone without package being updated; looks like there was an indirect dependency on MatrixStats, which has been updated since. /2023-06-14) > ☑︎ Fixed in Bioc devel > • GitHub: https://github.com/PhilBoileau/scPCA > • Notes: It uses functions from both matrixStats and DelayedMatrixStats > • Error: https://github.com/HenrikBengtsson/matrixStats/blob/58a8b54f571cb4fbae65b1bc4d738e2a2dd8e8b1/revdep/problems.md#scpca|https://github.com/HenrikBengtsson/matrixStats/blob/58a8b54f571cb4fbae65b1bc4d738e2a2dd8e8b1/revdep/problems.md#scpca > > Last updated: 2023-06-28 > > Regardless what the new default will be, these should be updated to explicitly set useNames = FALSE in the locations where useNames = TRUE break.

2023-07-05

Kevin Rue-Albrecht (16:29:42) (in thread): > > Instead, specify either useNames = TRUE or useNames = TRUE. > “We can do this the hard way… or the hard way.”:wink:

2023-07-10

Henrik Bengtsson (16:21:26) (in thread): > Woops. Fixed inhttps://github.com/HenrikBengtsson/matrixStats/commit/3e8f9ddafc0de61e870534b75fb5f90429a7bd31

2023-07-13

Alan Murphy (05:12:07): > Hey! I know the dev builds on kunpeng2 (Linux ARM64) are experimental but MungeSumstats is consistently failing because needed reference dataset packages aren’t installed: > > Package suggested but not available: 'SNPlocs.Hsapiens.dbSNP155.GRCh38' > > No problem if this is intentional (I know these packages are quite large) but thought I would mention it in case it is an oversight. Full list of large reference packages needed to test MSS are : > > SNPlocs.Hsapiens.dbSNP144.GRCh37, > SNPlocs.Hsapiens.dbSNP144.GRCh38 > SNPlocs.Hsapiens.dbSNP155.GRCh37 > SNPlocs.Hsapiens.dbSNP155.GRCh38 > BSgenome.Hsapiens.1000genomes.hs37d5 > BSgenome.Hsapiens.NCBI.GRCh38 >

Martin Grigorov (06:50:58) (in thread): > Let me take a look !

Martin Grigorov (06:52:32) (in thread): > some packages need to be installed manually and I still cannot understand why and which ones

Martin Grigorov (06:55:09) (in thread): > SNPlocs.Hsapiens.dbSNP155.GRCh38(almost 6GB) is being installed ! Once it is done I will install the others too !

Alan Murphy (07:29:52) (in thread): > Thank you! Might it not automatically installing be down to a timeout on the download? Whenever I install these I setoptions(timeout=2000)or something arbitrarily big

Martin Grigorov (07:31:13) (in thread): > BBS installs it and it uses this setting:https://github.com/Bioconductor/BBS/blob/devel/3.18/Renviron.bioc#L54

Martin Grigorov (07:36:16) (in thread): > but I’d expect a timeout error if this is the problem. Currently it just says that the package/dependency is not provided …

2023-07-18

Hervé Pagès (17:00:20) (in thread): > R CMD checkdoes not try to install any dep so you wouldn’t see a timeout forMungeSumstats. Furthermore, the daily report won’t report any details about the installation of deps, you’d have to look at the files in~/bioc-3.18-bioc/products-out/install/directly on kunpeng2 for that. What I’ve seen so far is that most of the time the package download gets corrupted/truncated on kunpeng2 for these big packages, not sure why.

2023-07-23

Sarvesh Nikumbh (07:31:13): > Hi! > Looks like the download stats for some packages are missing (all zeroes):http://bioconductor.org/packages/stats/bioc/seqArchR/http://bioconductor.org/packages/stats/bioc/seqArchRplus/

Lori Shepherd (15:52:32): > We’ll check and keep an eye out. We updated the system that generated the stats this last week so I expect it to be remedied this week on the next runs

2023-07-24

Lori Shepherd (11:59:40): > It looks like there is still an issue lingering. We are actively investigating and hope to have them back online shortly.

2023-07-28

Konstantinos Daniilidis (13:47:22): > @Konstantinos Daniilidis has joined the channel

2023-07-29

Lori Shepherd (10:26:50): > We are aware the download stats are still unavailable. We are continuing to investigate to make them available as soon as possible

2023-07-30

Sarvesh Nikumbh (10:38:29): > Thanks. I can see that the pages are back up, but the ranking is the other way round, e.g., Biostrings is ranked 2221/2230!:slightly_smiling_face:

Lori Shepherd (11:56:56): > Now that the stats are back, the ranks will regenerate as well over the next day

2023-08-03

Qiang Hu (12:02:47): > @Qiang Hu has joined the channel

2023-08-05

Qiang Hu (11:32:45): > Hi, two of my packagesRcwlandMACSrfailed because of failing to install their python dependencies. Could you please help to fix it? Thanks! > > # >>>>>>>>>>>>>>>>>>>>>> ERROR REPORT <<<<<<<<<<<<<<<<<<<<<< > > Traceback (most recent call last): > File "/home/biocbuild/.cache/R/basilisk/1.13.3/0/lib/python3.10/site-packages/conda/exception_handler.py", line 16, in *_call_* > return func(*args, **kwargs) > File "/home/biocbuild/.cache/R/basilisk/1.13.3/0/lib/python3.10/site-packages/conda/cli/main.py", line 84, in main_subshell > exit_code = do_call(args, p) > File "/home/biocbuild/.cache/R/basilisk/1.13.3/0/lib/python3.10/site-packages/conda/cli/conda_argparse.py", line 126, in do_call > return getattr(module, func_name)(args, parser) > File "/home/biocbuild/.cache/R/basilisk/1.13.3/0/lib/python3.10/site-packages/conda/notices/core.py", line 123, in wrapper > return func(*args, **kwargs) > File "/home/biocbuild/.cache/R/basilisk/1.13.3/0/lib/python3.10/site-packages/conda/cli/main_create.py", line 46, in execute > install(args, parser, "create") > File "/home/biocbuild/.cache/R/basilisk/1.13.3/0/lib/python3.10/site-packages/conda/cli/install.py", line 309, in install > unlink_link_transaction = solver.solve_for_transaction( > File "/home/biocbuild/.cache/R/basilisk/1.13.3/0/lib/python3.10/site-packages/conda/core/solve.py", line 153, in solve_for_transaction > unlink_precs, link_precs = self.solve_for_diff( > File "/home/biocbuild/.cache/R/basilisk/1.13.3/0/lib/python3.10/site-packages/conda/core/solve.py", line 214, in solve_for_diff > final_precs = self.solve_final_state( > File "/home/biocbuild/.cache/R/basilisk/1.13.3/0/lib/python3.10/site-packages/conda/core/solve.py", line 357, in solve_final_state > ssc = self._collect_all_metadata(ssc) > File "/home/biocbuild/.cache/R/basilisk/1.13.3/0/lib/python3.10/site-packages/conda/common/io.py", line 83, in decorated > return f(*args, **kwds) > File "/home/biocbuild/.cache/R/basilisk/1.13.3/0/lib/python3.10/site-packages/conda/core/solve.py", line 571, in _collect_all_metadata > index, r = self._prepare(prepared_specs) > File "/home/biocbuild/.cache/R/basilisk/1.13.3/0/lib/python3.10/site-packages/conda/core/solve.py", line 1285, in _prepare > reduced_index = get_reduced_index( > File "/home/biocbuild/.cache/R/basilisk/1.13.3/0/lib/python3.10/site-packages/conda/core/index.py", line 286, in get_reduced_index > new_records = SubdirData.query_all( > File "/home/biocbuild/.cache/R/basilisk/1.13.3/0/lib/python3.10/site-packages/conda/core/subdir_data.py", line 157, in query_all > result = tuple( > File "/home/biocbuild/.cache/R/basilisk/1.13.3/0/lib/python3.10/concurrent/futures/_base.py", line 621, in result_iterator > yield _result_or_cancel(fs.pop()) > File "/home/biocbuild/.cache/R/basilisk/1.13.3/0/lib/python3.10/concurrent/futures/_base.py", line 319, in _result_or_cancel > return fut.result(timeout) > File "/home/biocbuild/.cache/R/basilisk/1.13.3/0/lib/python3.10/concurrent/futures/_base.py", line 451, in result > return self.__get_result() > File "/home/biocbuild/.cache/R/basilisk/1.13.3/0/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result > raise self._exception > File "/home/biocbuild/.cache/R/basilisk/1.13.3/0/lib/python3.10/concurrent/futures/thread.py", line 58, in run > result = self.fn(*self.args, **self.kwargs) > File "/home/biocbuild/.cache/R/basilisk/1.13.3/0/lib/python3.10/site-packages/conda/core/subdir_data.py", line 142, in subdir_query > return tuple( > File "/home/biocbuild/.cache/R/basilisk/1.13.3/0/lib/python3.10/site-packages/conda/core/subdir_data.py", line 164, in query > self.load() > File "/home/biocbuild/.cache/R/basilisk/1.13.3/0/lib/python3.10/site-packages/conda/core/subdir_data.py", line 264, in load > _internal_state = self._load() > File "/home/biocbuild/.cache/R/basilisk/1.13.3/0/lib/python3.10/site-packages/conda/core/subdir_data.py", line 321, in _load > repodata, state = fetcher.fetch_latest_parsed() > File "/home/biocbuild/.cache/R/basilisk/1.13.3/0/lib/python3.10/site-packages/conda/gateways/repodata/*_init_*.py", line 757, in fetch_latest_parsed > parsed, state = self.fetch_latest() > File "/home/biocbuild/.cache/R/basilisk/1.13.3/0/lib/python3.10/site-packages/conda/gateways/repodata/*_init_*.py", line 819, in fetch_latest > cache.load_state() > File "/home/biocbuild/.cache/R/basilisk/1.13.3/0/lib/python3.10/site-packages/conda/gateways/repodata/*_init_*.py", line 625, in load_state > self.load(state_only=True) > File "/home/biocbuild/.cache/R/basilisk/1.13.3/0/lib/python3.10/site-packages/conda/gateways/repodata/*_init_*.py", line 575, in load > state = json.loads(state_file.read()) > File "/home/biocbuild/.cache/R/basilisk/1.13.3/0/lib/python3.10/json/*_init_*.py", line 346, in loads > return _default_decoder.decode(s) > File "/home/biocbuild/.cache/R/basilisk/1.13.3/0/lib/python3.10/json/decoder.py", line 340, in decode > raise JSONDecodeError("Extra data", s, end) > json.decoder.JSONDecodeError: Extra data: line 9 column 2 (char 283) > > `$ /home/biocbuild/.cache/R/basilisk/1.13.3/0/bin/conda create --yes --prefix /home/biocbuild/.cache/R/basilisk/1.13.3/Rcwl/1.17.1/env_Rcwl python=3.11 --quiet -c conda-forge` > > environment variables: > BBS_BIOC_MANIFEST_CLONE_PATH=/home/biocbuild/bbs-3.18-bioc/manifest > BBS_BIOC_VERSIONED_REPO_PATH=3.18/bioc > BBS_GITLOG_PATH=/home/biocbuild/bbs-3.18-bioc/gitlog > BBS_MEAT_PATH=/home/biocbuild/bbs-3.18-bioc/meat > BBS_REPORT_PATH=/home/biocbuild/public_html/BBS/3.18/bioc/report > CIO_TEST=<not set> > CONDA_DEFAULT_ENV=base > CONDA_EXE=/home/biocbuild/.cache/R/basilisk/1.13.3/0/bin/conda > CONDA_PREFIX=/home/biocbuild/.cache/R/basilisk/1.13.3/0 > CONDA_PROMPT_MODIFIER=(base) > CONDA_PYTHON_EXE=/home/biocbuild/.cache/R/basilisk/1.13.3/0/bin/python > CONDA_ROOT=/home/biocbuild/.cache/R/basilisk/1.13.3/0 > CONDA_SHLVL=1 > CURL_CA_BUNDLE=<not set> > LD_LIBRARY_PATH=/home/biocbuild/bbs-3.18-bioc/R/lib:/usr/local/lib:/usr/lib/x86_64- > linux-gnu:/usr/lib/jvm/java-11-openjdk-amd64/lib/server:/home/biocbuil > d/bbs-3.18-bioc/R/lib:/usr/local/lib:/usr/lib/x86_64-linux- > gnu:/usr/lib/jvm/java-11-openjdk-amd64/lib/server:/home/biocbuild/bbs- > 3.18-bioc/R/lib:/usr/local/lib:/usr/lib/x86_64-linux- > gnu:/usr/lib/jvm/java-11-openjdk-amd64/lib/server:/home/biocbuild/bbs- > 3.18-bioc/R/lib:/usr/local/lib:/usr/lib/x86_64-linux- > gnu:/usr/lib/jvm/java-11-openjdk-amd64/lib/server > LD_PRELOAD=<not set> > PATH=/home/biocbuild/.cache/R/basilisk/1.13.3/0/condabin:/home/biocbuild/.c > ache/R/basilisk/1.13.3/0/bin:/home/biocbuild/.cache/R/basilisk/1.13.3/ > 0/condabin:/home/biocbuild/.cache/R/basilisk/1.13.3/0/bin:/home/biocbu > ild/.cache/R/basilisk/1.13.3/0/bin:/home/biocbuild/.cache/R/basilisk/1 > .13.3/0/condabin:/home/biocbuild/bbs-3.18- > bioc/R/bin:/home/biocbuild/.local/bin:/home/biocbuild/bin:/usr/local/b > in:/usr/bin:/bin:/snap/bin:/home/biocbuild/.dotnet/tools:/usr/local/en > sembl-vep > PYTHONNOUSERSITE=1 > REQUESTS_CA_BUNDLE=<not set> > SSL_CERT_FILE=<not set> > > active environment : base > active env location : /home/biocbuild/.cache/R/basilisk/1.13.3/0 > shell level : 1 > user config file : /home/biocbuild/.condarc > populated config files : > conda version : 23.5.0 > conda-build version : not installed > python version : 3.10.12.final.0 > virtual packages : __archspec=1=x86_64 > __glibc=2.35=0 > __linux=5.15.0=0 > __unix=0=0 > base environment : /home/biocbuild/.cache/R/basilisk/1.13.3/0 (writable) > conda av data dir : /home/biocbuild/.cache/R/basilisk/1.13.3/0/etc/conda > conda av metadata url : None > channel URLs :[https://conda.anaconda.org/conda-forge/linux-64](https://conda.anaconda.org/conda-forge/linux-64)[https://conda.anaconda.org/conda-forge/noarch](https://conda.anaconda.org/conda-forge/noarch)[https://repo.anaconda.com/pkgs/main/linux-64](https://repo.anaconda.com/pkgs/main/linux-64)[https://repo.anaconda.com/pkgs/main/noarch](https://repo.anaconda.com/pkgs/main/noarch)[https://repo.anaconda.com/pkgs/r/linux-64](https://repo.anaconda.com/pkgs/r/linux-64)[https://repo.anaconda.com/pkgs/r/noarch](https://repo.anaconda.com/pkgs/r/noarch)package cache : /home/biocbuild/.cache/R/basilisk/1.13.3/0/pkgs > /home/biocbuild/.conda/pkgs > envs directories : /home/biocbuild/.cache/R/basilisk/1.13.3/0/envs > /home/biocbuild/.conda/envs > platform : linux-64 > user-agent : conda/23.5.0 requests/2.29.0 CPython/3.10.12 Linux/5.15.0-76-generic ubuntu/22.04.2 glibc/2.35 > UID:GID : 1002:1002 > netrc file : None > offline mode : False > > > An unexpected error has occurred. Conda has prepared the above report. >

2023-08-07

Jiaji George Chen (11:16:56): > @Jiaji George Chen has joined the channel

Andres Wokaty (11:46:25) (in thread): > I’ll take a look at these two. I wonder if it is because basilisk also erroring on devel?

Hervé Pagès (20:26:26) (in thread): > Can’t reproduce this on my Linux laptop (Ubuntu 23.04) using the most currentbasilisk(v 1.13.3, same as on the build machines). Maybe a cache issue on nebbiolo2? You could try to nuke~biocbuild/.cache/R/basiliskand reinstallbasiliskdirectly fromgit.bioconductor.orgsee if that helps?

Andres Wokaty (20:29:22) (in thread): > yeah, that’s what i am wondering. I am also not able to reproduce it in docker. I am going to remove this on nebbiolo2 and merida1. i am concerned it will still have the timeouts, though.

Hervé Pagès (21:31:00) (in thread): > Unlikely that it~won’t~will solvebasilisk’s timeouts on several platforms but that’s an issue with the package itself (unit tests take too long), not with the build machines. The maintainer can address this by moving more of the unit tests to the long tests.

2023-08-08

Kozo Nishida (02:05:20): > Hi BBS (Bioconductor Build System) users, > Is there a time lag between the BBS update and the devel webpage (+ the vignette) update? > I asked this becausehttp://bioconductor.org/checkResults/devel/bioc-LATEST/transomics2cytoscape/has been updated buthttp://bioconductor.org/packages/3.18/bioc/html/transomics2cytoscape.htmlhas not yet been updated. - Attachment (Bioconductor): transomics2cytoscape (development version) > transomics2cytoscape generates a file for 3D transomics visualization by providing input that specifies the IDs of multiple KEGG pathway layers, their corresponding Z-axis heights, and an input that represents the edges between the pathway layers. The edges are used, for example, to describe the relationships between kinase on a pathway and enzyme on another pathway. This package automates creation of a transomics network as shown in the figure in Yugi.2014 (https://doi.org/10.1016/j.celrep.2014.07.021) using Cytoscape automation (https://doi.org/10.1186/s13059-019-1758-4).

Lori Shepherd (07:37:05): > Yes there is normally a delay; normally a few hours up to 24 depending on when the builds finish and send information

Qiang Hu (10:33:19) (in thread): > Thanks for looking this problem. It failed in the beginning when trying to install python modules not in testing step. Thanks!

Andres Wokaty (11:10:22) (in thread): > Removing the cache on nebbiolo2 allows me to manually runR CMD build Rcwlwithout issue.

Hervé Pagès (14:41:13) (in thread): > > It failed in the beginning when trying to install python modules not in testing step. > My comment was about the unit tests taking too long in thebasiliskpackage, not in your packages@Qiang Hu

2023-08-11

Jiefei Wang (16:55:40): > @Jiefei Wang has joined the channel

2023-08-14

Kozo Nishida (04:17:42) (in thread): > I think the vignette’s webpage isn’t being updated after all. > * http://bioconductor.org/checkResults/devel/bioc-LATEST/transomics2cytoscape/ > * http://bioconductor.org/packages/3.18/bioc/vignettes/transomics2cytoscape/inst/doc/transomics2cytoscape.html > Let me know if you have any ideas.

Andres Wokaty (10:39:23) (in thread): > I’ll look into this.

Tyrone Lee (12:53:17): > @Tyrone Lee has joined the channel

Hervé Pagès (13:45:55) (in thread): > transomics2cytoscape1.11.0 was published on April 26 as part of BioC 3.18, when we released BioC 3.17. The day prior to the BioC 3.17 release, we bumped the version of the package and created the RELEASE_3_17 branch. Right after the version bumps, the version was 1.10.0 in release and 1.11.0 in devel. > Right now, the version is still 1.11.0 in git for the devel branch so it won’t replace whatever was pusblished in BioC 3.18 on April 26 unless you bump the version of the package to 1.11.1.

Kozo Nishida (16:41:37) (in thread): > @Hervé PagèsThanks for the info! That must be it.:pray:@Lori Shepherd@Andres WokatyI’m sorry for disturbing you.:pray:

2023-08-20

Federica Gazzelloni (10:38:01): > @Federica Gazzelloni has joined the channel

2023-08-24

Lachlan Baer (01:20:40): > @Lachlan Baer has joined the channel

2023-08-30

Sagun Maharjan (13:08:22): > @Sagun Maharjan has joined the channel

Sagun Maharjan (13:24:41): > Hi all, > We have been seeing windows build error for Macarron package, in****palomino3******** specific. Any recommendation of getting the build issue fixed? ****http://bioconductor.org/checkResults/release/bioc-LATEST/Macarron/palomino3-checksrc.html

Hervé Pagès (15:46:52) (in thread): > Seems like the error has been around for a while (several months?). I suspect that your little internal helper functionChemTax_HMDB()(used insidedecorateID()) fails on Windows for reasons that would require some investigation. Do you have access to a Windows machine? DoesChemTax_HMDB("HMDB0000874")work as expected on it? > > Here is howChemTax_HMDB()is implemented: > > ChemTax_HMDB <- function(h){ > hmdb_url <- paste0("[https://hmdb.ca/metabolites/](https://hmdb.ca/metabolites/)",h,".xml") > Sys.sleep(0.1) > if(RCurl::url.exists(hmdb_url)){ > hmdb_page <- xml2::read_xml(hmdb_url) > node_class <- xml2::xml_text(xml2::xml_find_all(hmdb_page, "//class")) > node_subclass <- xml2::xml_text(xml2::xml_find_all(hmdb_page, "//sub_class")) > cbind(h, node_subclass, node_class) > } > } > > So if the URL does not exist, the function does nothing and returns an invisibleNULL. Unfortunately, many obscure downstream errors will likely result of that, and they will probably be hard to understand. > It’s always better to fail early and loudly, and with a clear error message. Doing so is not only more user-friendly but it will also make it easier for you to understand why your code fails on systems that you don’t necessarily have access to.

2023-09-01

Jacques SERIZAY (07:28:41): > Hi all, I have issues when building vignettes/examples fromHiCExperiment,HiContactsandHiCoolpackages,specifically on the ****release******** version and ********nebbiolo1****, although the code is the same onreleaseanddevel. I cannot seem to replicate this usingbioconductor_docker:release, so I suspect a connection error withExperimentHubThe errors are allProvided file is not a .cool/.mcool file, and occur in the first chunk/example that makes use ofHiCExperiment::contacts_yeast()(dochere). This function is used in documentation to generate a toy example, and internally callsHiContactsData::HiContactsData('yeast_wt', 'cool')which fetches a.coolfile (EH7702) from theExperimentHub. The error messages likely occur when theimportfunction is ran on the file cached byHiContactsData. This suggests that the cached file is not compatible with theimport. But when I check locally with the different dockers,HiCExperiment::contacts_yeast()andimportwork ok. > > I don’t exactly know how to investigate this issue. It’s also puzzling that it occurs specifically onreleaseandnebbiolo1but notdevelor other machines. Would anyone have any suggestion on how to investigate/solve this issue? - Attachment (jserizay.com): Example datasets provided in HiCExperiment & HiContactsData — data > Example datasets provided in HiCExperiment & HiContactsData

2023-09-05

Lori Shepherd (08:21:41): > The release schedule for Bioconductor release 3.18 can be found at:https://bioconductor.org/developers/release-schedule/. Please be mindful of important deadlines.

Sagun Maharjan (13:57:16) (in thread): > Hi@Hervé Pagès, > Thank you so much for your comments. I will try to debug the function in a windows machine and hopefully resolve the issue.

2023-09-06

Mike Smith (07:13:08): > There’s a number of packages that currently fail to build on the Mac x64 builders (Merida and lconway) because they can’t find the headers for the GNU scientific library e.g. > * PICS > * GLAD > * immunoClust > The symptons seem to be the same for all of them, and I wonder if it’s possible to get the GSL headers installed on that system and restore the build process. The packages seem to be fine on other platforms, including the Mac arm64 builder.

Andres Wokaty (09:37:54) (in thread): > Thanks for reporting this. GLS is installed and we should see the errors resolve in the build report tomorrow.

Mike Smith (09:39:20) (in thread): > Awesome thanks. I think this will then restore some of their dependents too.

2023-09-12

Y-h. Taguchi (20:32:33): > I am not sure if it is a suitable place to ask this. > I have had a build error on my package TDbasedUFEadv. Build error messages arehttps://bioconductor.org/checkResults/devel/bioc-LATEST/TDbasedUFEadv/nebbiolo2-buildsrc.htmlhttps://bioconductor.org/checkResults/devel/bioc-LATEST/TDbasedUFEadv/lconway-buildsrc.htmlhttps://bioconductor.org/checkResults/devel/bioc-LATEST/TDbasedUFEadv/kunpeng2-buildsrc.htmlAll of these said that > “cannot open URL ‘https://stringdb-downloads.org/download/protein.aliases.v11.5/9606.protein.aliases.v11.5.txt.gz’” > But I can open this URL from my computer without any problem. > This URL is embedded in one of packages I include. Thus I cannot touch the code to use this URL at all. > > I am not sure if it can be the reason that TDbasedUFEadv is not included into BioC 3.18. I am glad if you can advice me on this point. Since this seems to be intermittent problem, I hope that it cannot be the reason that TDbasedUFEadv is not included in Bioc 3.18.

2023-09-13

Martin Grigorov (02:46:26) (in thread): > I can check on kunpeng2 !

Martin Grigorov (02:48:45) (in thread): > wgethttps://stringdb-downloads.org/download/protein.aliases.v11.5/9606.protein.aliases.v11.5.txt.gzworks ! But the speed is slow - 8Kb/sec. It would take around 35 mins to download it

Martin Grigorov (02:49:14) (in thread): > the machine is in Singapore

Martin Grigorov (02:49:43) (in thread): > I also tried it from my dev machine (Bulgaria, Europe) and the file has been downloaded in few secs

Y-h. Taguchi (02:52:56) (in thread): > @Martin GrigorovThank you very much for your confirmation. I need to include the package that use this URL definitely and the package is of course included in CRAN or Bioc. Thus I expect that using this package cannot be a reason not to be included into Bioc 3.18 even if build fails because of this package. Am I right?

Martin Grigorov (02:54:23) (in thread): > I am not a member of the Bioc core team and I cannot give you an answer on this

Y-h. Taguchi (02:55:45) (in thread): > @Martin GrigorovThanks. Where can I ask this? Are there anyone who belongs to Bioc core team in this channel or must I ask it at other channel?

Martin Grigorov (02:55:53) (in thread): > which package provides this file ?

Y-h. Taguchi (02:56:07) (in thread): > StringDB

Martin Grigorov (02:56:41) (in thread): > I believe everyone from the core team is here. Just give them time to wake up. I believe all of them are based in North America

Y-h. Taguchi (02:57:17) (in thread): > @Martin GrigorovI see. Thanks. Then I am waiting for their reply.

Vince Carey (06:14:10) (in thread): > Thanks@Martin Grigorovfor investigating.@Y-h. Taguchias far as I can tell that was an intermittent problem with the protein aliases server; the file was retrieved quickly when I attempted this morning. If the problem recurs you can dialogue with the maintainer of the troublesome package to help improve its robustness. There is no need at this time to be concerned about omission from 3.18.

Y-h. Taguchi (07:25:04) (in thread): > @Vince CareyI am very pleased to know that it cannot be the reason to be omitted from Bioc 3.18. Thank you very much.

2023-09-15

Ludwig Geistlinger (16:29:36): > There seems to be a problem with the download stats, see eghttps://bioconductor.org/packages/stats/bioc/GenomicRanges/andhttps://bioconductor.org/packages/stats/bioc/SummarizedExperiment/

Lori Shepherd (17:11:57): > @Hervé Pagèscan confirm but there was an issue running this last run and it should resolve on tht next run

Hervé Pagès (20:53:48): > Yeah the stats are down again, sorry for that. It will take about 48 hrs for them to come back.

2023-09-17

Hervé Pagès (21:09:46): > Another unexpected issue went in the way so it will take a few more days for the stats to self-repair. Sorry for the inconvenience.

2023-09-19

Laurent Gatto (08:01:11) (in thread): > Hi Hervé - any idea of how much ‘a few more’ will be? I need these data for a document for a grant proposal that needs to be submitted by Thursday.

Laurent Gatto (08:05:52) (in thread): > @Marcel Ramos Pérez(who happens to be in Belgium for the EuroBioc workshops and conference!) has some locally cached download stats that will address my needs. Thanks Marcel!

Hervé Pagès (13:57:09) (in thread): > Really sorry for the inconvenience Laurent. It might still take about 24hrs before the stats are back online. Glad that Marcel was able to help you with this.

2023-09-20

Hervé Pagès (20:39:37) (in thread): > The stats are finally back:https://bioconductor.org/packages/stats/Software packages only! The stats for data-annotation/data-exp/workflow packages are available but lagging several days behind, because I temporarily stopped them to focus on the software stats. Now they are on again and will update in the next few days:crossed_fingers:Thanks again for your patience.

2023-09-21

Philippine Louail (16:40:16): > @Philippine Louail has joined the channel

2023-09-29

Dima Lvovs (12:26:52): > @Dima Lvovs has joined the channel

Dima Lvovs (12:33:06): > Hello, everyone, a bit puzzled here, maybe someone may direct me? How come a package check fails on some OSes and not fail on the others, especially when the error is the same, and all OSes where it does fail have the required packages installed: > failed check innebbiolo2 > > Error: processing vignette 'CoGAPS.Rmd' failed with diagnostics: > unable to find required package 'SeuratObject' > > installed innebbiolo2 > > Seurat /home/biocbuild/bbs-3.18-bioc/R/site-library 4.4.0 4.3.1 > SeuratObject /home/biocbuild/bbs-3.18-bioc/R/site-library 4.1.4 4.3.1 >

Lori Shepherd (12:34:42): > If its only on linux please see related announcements:https://stat.ethz.ch/pipermail/bioc-devel/2023-May/019692.html

Dima Lvovs (12:35:27) (in thread): > oh thank you so much!

2023-10-04

Amanda Hiser (09:44:40): > @Amanda Hiser has joined the channel

2023-10-08

Peter Hickey (21:32:07): > What are the build reports inhttps://bioconductor.org/checkResults/3.18/bioc-testing-LATEST/? As a developer, do I need to worry about these ‘testing’ builds?

2023-10-09

Y-h. Taguchi (04:37:43): > Deer people, > > Thanks for your every effort for maintaining Bioc. By the way, > TDbasedUFEadv (development version ) still has intermittent problems,https://bioconductor.org/packages/devel/bioc/html/TDbasedUFEadv.htmlIt can be sometimes successful but not time to time without any updates in my side. I think that it cannot be the reason not to be included in Bioc 3.18. > I am glad if someone can confirm this point:sweat_smile: - Attachment (Bioconductor): TDbasedUFEadv (development version) > This is an advanced version of TDbasedUFE, which is a comprehensive package to perform Tensor decomposition based unsupervised feature extraction. In contrast to TDbasedUFE which can perform simple the feature selection and the multiomics analyses, this package can perform more complicated and advanced features, but they are not so popularly required. Only users who require more specific features can make use of its functionality.

Hervé Pagès (11:46:25) (in thread): > This is experimental stuff. Please ignore.

Hervé Pagès (12:07:58) (in thread): > Aren’t these intermittent problems caused by some known flakiness with thestringdb-downloads.orgserver? Looks like you already asked about this herehttps://community-bioc.slack.com/archives/CEQ04GKEC/p1694565153539729, and that@Vince Careyalready answered you w.r.t. inclusion of the package in BioC 3.18.

Y-h. Taguchi (16:11:05) (in thread): > @Hervé PagèsThank you very much for your quick reply. It is not only download problem, but other issues on compiling vignette, which is an intermittent problem and was reported previously, too, which I cannot resolve the reason. > Now it has happened not in development packaged but in released packagehttps://bioconductor.org/packages/release/bioc/html/TDbasedUFEadv.html(Since it is intermittent, it has vanished in the developement version) - Attachment (Bioconductor): TDbasedUFEadv > This is an advanced version of TDbasedUFE, which is a comprehensive package to perform Tensor decomposition based unsupervised feature extraction. In contrast to TDbasedUFE which can perform simple the feature selection and the multiomics analyses, this package can perform more complicated and advanced features, but they are not so popularly required. Only users who require more specific features can make use of its functionality.

Peter Hickey (16:39:11) (in thread): > Thanks!

Vince Carey (17:07:46) (in thread): > If you would like to avoid the intermittent error states, why not write your code conditionally on success at contacting the server? You could have some mock data to use when the server is down. The mock would be used when server is down and the real data would be used when the server is up.

Hervé Pagès (17:37:43) (in thread): > @Y-h. TaguchiThe intermittent vignette TIMEOUTs are not new and were discussed here 6 months ago:https://community-bioc.slack.com/archives/C35G93GJH/p1680992780183569

Y-h. Taguchi (17:43:41) (in thread): > @Vince CareyThank you very much for your comment. Intermittent problem is not related to server as@Hervé Pagèspointed out.@Hervé PagèsThank you very much for your comment. Yes, it is not new, but I have received automatic warning again recently. Is it still not related to inclusion into future release? Is it possible for you to suppress the warning about intermittent problem? It is a little bit confusing.

Hervé Pagès (18:22:05) (in thread): > Unfortunately, with 2000+ packages in the daily builds, it’s not realistic that we start to manually maintain a list of exceptions. But also, silencing the problem is usually not the right thing to do, especially when the problem is not fully understood (like in this case). > At some point, someone will need to go to the bottom of why the vignette has these intermittent ERRORs or TIMEOUTs on Linux. Note that we’ve seen them on the Linux intel builders that we maintain but also on the Linux arm64 builder maintained by@Martin Grigorov. The fact that the ERROR happens right after downloading a file fromstringdb-downloads.orgstrongly suggests that the issue is related with the download. Hard to know exactly what, but how can you dismiss entirely the possibility that the problem is onstringdb-downloads.org’s side? > As already discussed 6 months ago, you could try to reproduce this by using our docker image (Linux intel), or by following@Martin Grigorovinstructions herehttps://blog.bioconductor.org/posts/2023-06-09-debug-linux-arm64-on-docker/if you want to do this on Linux arm64. Note that, since the problem is intermittent, the fact that your vignette will evaluate without problem the first time you try it won’t be enough to draw any conclusion. You might need to try this at different times of the day, different days of the week. You could try to automate this by using a cronjob to evaluate your vignette automatically several times per day for a few days, and log the output. If that doesn’t work, you could try to make the code in the vignette more verbose. This would typically be achieved by adding averboeargument (FALSEby default) to some of your functions, and set this argument toTRUEwhen calling these functions in the vignette. When executed in verbose mode, your code would print its progress and other useful information about what it’s doing. Then, when the ERROR happens during the daily builds, the report will display all this information. This will hopefully help understand the problem. Using the verbose mode in the vignette would be temporary of course, until more insight is achieved. > I fully acknowledge that troubleshooting this sort of intermittent problem is not easy and can be time consuming, but the burden is ultimately on the developer/maintainer of the package. - Attachment (blog.bioconductor.org): Bioconductor community blog - Emulated build and test of Bioconductor packages for Linux ARM64 > Build and test for Linux ARM64 with Docker on x86_64 host

Y-h. Taguchi (19:12:26) (in thread): > @Hervé PagèsThank you very much for your comments. I understand that it is very difficult to tackle this problem and waring will not stop. Can you still include TDbasedUFEadv into Bioc 3.18? I will try to fix intermittent problem in next six months. At the moment, I have no time to fix intermittent problem till the release of Bioc 3.18.

Hervé Pagès (21:06:00) (in thread): > As explained already, these intermittent failures do not prevent the package from being included in the next release. No special action required on our side.

Y-h. Taguchi (21:07:18) (in thread): > @Hervé PagèsThanks. But I will try to fix it as soon as possible in the next six months

2023-10-10

Martin Grigorov (02:44:26) (in thread): > Hi! Could something be improved in the traceability/logging related to the TIMEOUT_** settings at BBS level, i.e. globally for all packages ? E.g. dump the process/thread stack trace of the current operation before killing it due to timeout.https://stat.ethz.ch/R-manual/R-devel/library/utils/html/debugger.htmlhttps://renkun.me/2020/03/31/a-simple-way-to-show-stack-trace-on-error-in-r/

Hervé Pagès (13:19:11) (in thread): > This is probably much harder than it sounds. PR welcome!

2023-10-11

Steffen Neumann (07:25:46): > Hi@Martin Grigorovand@Yikun Jiangamong the failling packages on kunpeng2 is also xcms,https://bioconductor.org/checkResults/devel/bioc-LATEST/xcms/kunpeng2-buildsrc.htmlwith error message ERROR: lazy loading failed for package ‘xcms’ > and I don’t find more verbose hints about the underlying cause.

Steffen Neumann (07:25:56) (in thread): > Any further diagnostics I could check ?

Martin Grigorov (07:28:09) (in thread): > Hi@Steffen Neumann! Let me see !

Martin Grigorov (07:40:25) (in thread): > > $ R CMD build xcms > * checking for file 'xcms/DESCRIPTION' ... OK > * preparing 'xcms': > * checking DESCRIPTION meta-information ... OK > * cleaning src > * installing the package to build vignettes > * creating vignettes ... OK > * cleaning src > * checking for LF line-endings in source and make files and shell scripts > * checking for empty or unneeded directories > * looking to see if a 'data/datalist' file should be added > * building 'xcms_3.99.5.tar.gz' > > it seems BBS does something more …

Martin Grigorov (07:48:25) (in thread): > > biocbuild@kunpeng2 ~/git> /home/biocbuild/R/R-4.3.1/bin/R CMD build --keep-empty-dirs --no-resave-data xcms > * checking for file 'xcms/DESCRIPTION' ... OK > * preparing 'xcms': > * checking DESCRIPTION meta-information ... OK > * cleaning src > * installing the package to build vignettes > * creating vignettes ... ERROR > --- re-building 'LC-MS-feature-grouping.Rmd' using rmarkdown > --- finished re-building 'LC-MS-feature-grouping.Rmd' > > --- re-building 'xcms-direct-injection.Rmd' using rmarkdown > --- finished re-building 'xcms-direct-injection.Rmd' > > --- re-building 'xcms-lcms-ms.Rmd' using rmarkdown > --- finished re-building 'xcms-lcms-ms.Rmd' > > --- re-building 'xcms.Rmd' using rmarkdown > biocbuild@kunpeng2 ~/git [1]> > > an error, but a different one than the report

Martin Grigorov (07:52:54) (in thread): > > > BiocManager::install(c("xcms"), type = "source", checkBuilt = TRUE, force = TRUE) > Bioconductor version 3.18 (BiocManager 1.30.22), R 4.3.1 (2023-06-16) > Installing package(s) 'xcms' > trying URL '[https://bioconductor.org/packages/3.18/bioc/src/contrib/xcms_3.99.5.tar.gz](https://bioconductor.org/packages/3.18/bioc/src/contrib/xcms_3.99.5.tar.gz)' > Content type 'application/x-gzip' length 8333250 bytes (7.9 MB) > ================================================== > downloaded 7.9 MB > > * installing **source** package 'xcms' ... > **** using staged installation > **** libs > using C compiler: 'gcc (GCC) 10.3.1' > using C++ compiler: 'g++ (GCC) 10.3.1' > rm -f massifquant/xcms_massifquant.o massifquant/TrMgr.o massifquant/Tracker.o massifquant/SegProc.o massifquant/DataKeeper.o massifquant/OpOverload.o obiwarp/mat.o obiwarp/vec.o obiwarp/xcms_dynprog.o obiwarp/xcms_lmat.o xcms_obiwarp.o fastMatch.o mzClust_hclust.o mzROI.o util.o xcms.o binners.o > g++ -std=gnu++17 -I"/home/biocbuild/R/R-4.3.1/include" -DNDEBUG -I/usr/local/include -fPIC -g -O2 -Wall -c massifquant/xcms_massifquant.cpp -o massifquant/xcms_massifquant.o > g++ -std=gnu++17 -I"/home/biocbuild/R/R-4.3.1/include" -DNDEBUG -I/usr/local/include -fPIC -g -O2 -Wall -c massifquant/TrMgr.cpp -o massifquant/TrMgr.o > g++ -std=gnu++17 -I"/home/biocbuild/R/R-4.3.1/include" -DNDEBUG -I/usr/local/include -fPIC -g -O2 -Wall -c massifquant/Tracker.cpp -o massifquant/Tracker.o > g++ -std=gnu++17 -I"/home/biocbuild/R/R-4.3.1/include" -DNDEBUG -I/usr/local/include -fPIC -g -O2 -Wall -c massifquant/SegProc.cpp -o massifquant/SegProc.o > g++ -std=gnu++17 -I"/home/biocbuild/R/R-4.3.1/include" -DNDEBUG -I/usr/local/include -fPIC -g -O2 -Wall -c massifquant/DataKeeper.cpp -o massifquant/DataKeeper.o > g++ -std=gnu++17 -I"/home/biocbuild/R/R-4.3.1/include" -DNDEBUG -I/usr/local/include -fPIC -g -O2 -Wall -c massifquant/OpOverload.cpp -o massifquant/OpOverload.o > g++ -std=gnu++17 -I"/home/biocbuild/R/R-4.3.1/include" -DNDEBUG -I/usr/local/include -fPIC -g -O2 -Wall -c obiwarp/mat.cpp -o obiwarp/mat.o > g++ -std=gnu++17 -I"/home/biocbuild/R/R-4.3.1/include" -DNDEBUG -I/usr/local/include -fPIC -g -O2 -Wall -c obiwarp/vec.cpp -o obiwarp/vec.o > g++ -std=gnu++17 -I"/home/biocbuild/R/R-4.3.1/include" -DNDEBUG -I/usr/local/include -fPIC -g -O2 -Wall -c obiwarp/xcms_dynprog.cpp -o obiwarp/xcms_dynprog.o > obiwarp/xcms_dynprog.cpp: In member function 'void DynProg::find_path(VEC::MatF&, VEC::VecF&, int, float, float, int, float)': > obiwarp/xcms_dynprog.cpp:1113:9: warning: variable 'bestscore' set but not used [-Wunused-but-set-variable] > 1113 | float bestscore; > | ^~~~~~~~~~~~~~~~~ > g++ -std=gnu++17 -I"/home/biocbuild/R/R-4.3.1/include" -DNDEBUG -I/usr/local/include -fPIC -g -O2 -Wall -c obiwarp/xcms_lmat.cpp -o obiwarp/xcms_lmat.o > g++ -std=gnu++17 -I"/home/biocbuild/R/R-4.3.1/include" -DNDEBUG -I/usr/local/include -fPIC -g -O2 -Wall -c xcms_obiwarp.cpp -o xcms_obiwarp.o > gcc -I"/home/biocbuild/R/R-4.3.1/include" -DNDEBUG -I/usr/local/include -fPIC -g -O2 -Wall -c fastMatch.c -o fastMatch.o > gcc -I"/home/biocbuild/R/R-4.3.1/include" -DNDEBUG -I/usr/local/include -fPIC -g -O2 -Wall -c mzClust_hclust.c -o mzClust_hclust.o > gcc -I"/home/biocbuild/R/R-4.3.1/include" -DNDEBUG -I/usr/local/include -fPIC -g -O2 -Wall -c mzROI.c -o mzROI.o > gcc -I"/home/biocbuild/R/R-4.3.1/include" -DNDEBUG -I/usr/local/include -fPIC -g -O2 -Wall -c util.c -o util.o > gcc -I"/home/biocbuild/R/R-4.3.1/include" -DNDEBUG -I/usr/local/include -fPIC -g -O2 -Wall -c xcms.c -o xcms.o > gcc -I"/home/biocbuild/R/R-4.3.1/include" -DNDEBUG -I/usr/local/include -fPIC -g -O2 -Wall -c binners.c -o binners.o > binners.c: In function '_breaks_on_binSize': > binners.c:357:7: warning: unused variable 'idx' [-Wunused-variable] > 357 | int idx = 0; > | ^~~~~ > g++ -std=gnu++17 -shared -L/home/biocbuild/R/R-4.3.1/lib -L/usr/local/lib -o xcms.so massifquant/xcms_massifquant.o massifquant/TrMgr.o massifquant/Tracker.o massifquant/SegProc.o massifquant/DataKeeper.o massifquant/OpOverload.o obiwarp/mat.o obiwarp/vec.o obiwarp/xcms_dynprog.o obiwarp/xcms_lmat.o xcms_obiwarp.o fastMatch.o mzClust_hclust.o mzROI.o util.o xcms.o binners.o -L/home/biocbuild/R/R-4.3.1/lib -lR > installing to /home/biocbuild/R/R-4.3.1/site-library/00LOCK-xcms/00new/xcms/libs > **** R > **** data > **** inst > **** byte-compile and prepare package for lazy loading > Creating a new generic function for 'group' in package 'xcms' > Creating a new generic function for 'sigma' in package 'xcms' > **** help > ***** installing help indices > ***** copying figures > **** building package indices > **** installing vignettes > **** testing if installed package can be loaded from temporary location > **** checking absolute paths in shared objects and dynamic libraries > **** testing if installed package can be loaded from final location > **** testing if installed package keeps a record of temporary installation path > * DONE (xcms) > > The downloaded source packages are in > '/home/biocbuild/tmp/RtmprKjg3H/downloaded_packages' > > now it looks like in the report but there is no problem

Martin Grigorov (07:54:04) (in thread): > I have the feeling that the next report will be green

Peter Hickey (17:05:39): > Another one for@Martin Grigorovand@Yikun Jiang(thanks for making it easy to emulate building and check on Linux ARM64) > I’m trying to make sense of theR CMD checkerror on kunpeng2 forDelayedMatrixStats(https://bioconductor.org/checkResults/3.18/bioc-LATEST/DelayedMatrixStats/kunpeng2-checksrc.html). > I started by trying to reproduce using the Docker image following instructions inhttps://blog.bioconductor.org/posts/2023-06-09-debug-linux-arm64-on-docker/RunningR CMD check DelayedMatrixStats_1.23.5.tar.gzI first hit a different error related to checking the PDF version of the manual > > # Output from Docker image > * checking PDF version of manual ... WARNING > LaTeX errors when creating PDF version. > This typically indicates Rd problems. > * checking PDF version of manual without index ... ERROR > Re-running with no redirection of stdout/stderr. > Hmm ... looks like a package > Converting parsed Rd's to LaTeX .. > Creating pdf output from LaTeX ... > Warning in texi2dvi(file = file, pdf = TRUE, clean = clean, quiet = quiet, : > texi2dvi script/program not available, using emulation > Error in texi2dvi(file = file, pdf = TRUE, clean = clean, quiet = quiet, : > pdflatex is not available > Warning in texi2dvi(file = file, pdf = TRUE, clean = clean, quiet = quiet, : > texi2dvi script/program not available, using emulation > Error in texi2dvi(file = file, pdf = TRUE, clean = clean, quiet = quiet, : > pdflatex is not available > Error in running tools::texi2pdf() > > To workaround the above, since the error on kunpeng2 happened while running the tests, I instead rantestthat::test_local()from the package source directory. > All of the tests passed. > > Looking closer at the error on kunpeng2, it looks like something other than the tests actually failing … perhaps the process was killed for reasons I can’t see or make sense of? > > # Output on kunpeng2 > * checking tests ... > Running ‘testthat.R’/home/biocbuild/R/R-4.3.1/bin/BATCH: line 60: 1293792 Killed ${R_HOME}/bin/R -f ${in} ${opts} ${R_BATCH_OPTIONS} > ${out} 2>&1 > > ERROR > Running the tests in ‘tests/testthat.R’ failed. > Last 13 lines of output: > > The following objects are masked from 'package:matrixStats': > > colAnyMissings, rowAnyMissings > > > > > test_check("DelayedMatrixStats") > Loading required package: rhdf5 > > Attaching package: 'HDF5Array' > > The following object is masked from 'package:rhdf5': > > h5ls >

Hervé Pagès (20:08:52) (in thread): > Note that today’s report shows an abnormally high number of TIMEOUTs and ERRORs for kunpeng2. (Seehttps://bioconductor.org/checkResults/devel/bioc-LATEST/kunpeng2-index.html) So it seems that something happened to the machine that affected its ability to build and check packages. - File (PNG): Screenshot from 2023-10-11 17-06-43.png

Hervé Pagès (20:11:21) (in thread): > Seehttps://community-bioc.slack.com/archives/CEQ04GKEC/p1697069332979499?thread_ts=1697023546.334579&cid=CEQ04GKECMaybe the problem will go away tomorrow:crossed_fingers: - Attachment: Attachment > Note that today’s report shows an abnormally high number of TIMEOUTs and ERRORs for kunpeng2. (See https://bioconductor.org/checkResults/devel/bioc-LATEST/kunpeng2-index.html) So it seems that something happened to the machine that affected its ability to build and check packages.

2023-10-12

Martin Grigorov (02:05:22) (in thread): > I will build/check it manually now

Martin Grigorov (02:08:27) (in thread): > I wonder whether tweaking the settings would help -https://github.com/Bioconductor/BBS/blob/537713f8ef6085b57249078d34aa51b53c28ebb5/3.18/bioc/kunpeng2/config.sh#L15-L17htopshows that the system is quite loaded. I have the feeling this might be the reason why some packages timeout

Martin Grigorov (02:22:50) (in thread): > > ... > * checking PDF version of manual ... OK > * DONE > > Status: 1 NOTE > See > '/home/biocbuild/git/DelayedMatrixStats.Rcheck/00check.log' > for details. > > all is good

Martin Grigorov (02:26:54) (in thread): > changed them to > > export BBS_NB_CPU=25 # 32 cores are available > export BBS_BUILD_NB_CPU=20 # 32 cores are available > export BBS_CHECK_NB_CPU=20 # 32 cores are available >

Martin Grigorov (03:36:38) (in thread): > today’sR CMD check xcmspasses: > > ... > * checking tests ... > Running 'testthat.R' > OK > * checking for unstated dependencies in vignettes ... OK > * checking package vignettes in 'inst/doc' ... OK > * checking running R code from vignettes ... > 'LC-MS-feature-grouping.Rmd' using 'UTF-8'... OK > 'xcms-direct-injection.Rmd' using 'UTF-8'... OK > 'xcms-lcms-ms.Rmd' using 'UTF-8'... OK > 'xcms.Rmd' using 'UTF-8'... OK > NONE > * checking re-building of vignette outputs ... OK > * checking PDF version of manual ... OK > * DONE > > Status: 5 NOTEs > See > '/home/biocbuild/git/xcms.Rcheck/00check.log' > for details. >

Mike Smith (03:44:38) (in thread): > @Peter HickeyI think your initial issue with the missingpdflatexis because the Bioc docker container doesn’t come with a latex distribution installed. I think I ran into this same issue (https://blog.bioconductor.org/posts/2023-07-14-linux-arm64-github-actions/#choice-of-docker-container) and have been building an arm64 container with Tinytex installed to work around it. You can find that athttps://github.com/grimbough/bioc-testing-with-arm64/pkgs/container/bioc-with-tinytexif it’s useful to you.

Martin Grigorov (03:46:25) (in thread): > yes! the other day someone had the same problem in the mailing list. To installpdflatexyou can doapt install texlive-latex-extra

Mike Smith (04:02:48) (in thread): > Yep, that works too. I found the combination of using GHA and installing anything extra pretty slow under emulation, hence why I went for the pre-built image, but your approach is probably easier if you’re working in a local docker container.

Maximilian Mattessich (09:31:17): > Sorry to pile on regarding kunpeng2, but our packagenipalsMCIAalso generates an ERROR while building vignettes only on kunpeng2 (link). The logs do not show any reason why this error occurred. > > Is it possible to get more verbose output from the BUILD log without emulating Linux ARM64 ourselves?

Peter Hickey (17:35:49) (in thread): > Thanks@Mike Smithand@Martin Grigorov. For whatever reason I always get confounded when using Docker when really I just need to recall that I can install software like on other Linux machines I use!

Hervé Pagès (19:09:37) (in thread): > @Martin GrigorovAbsolutely! Seehttps://github.com/Bioconductor/BBS/pull/293#issuecomment-1579151180Want to submit a PR for this? Thanks!

2023-10-13

Martin Grigorov (02:01:43) (in thread): > https://bioconductor.org/checkResults/devel/bioc-LATEST/nipalsMCIA/is fully green today!:slightly_smiling_face:

Martin Grigorov (02:02:42) (in thread): > https://bioconductor.org/checkResults/3.18/bioc-LATEST/DelayedMatrixStats/kunpeng2-checksrc.html- kunpeng2 has only check warnings today! Like the other builders.

Martin Grigorov (02:03:45) (in thread): > https://bioconductor.org/checkResults/devel/bioc-LATEST/xcms/kunpeng2-buildsrc.htmlis fully green today! > I will open a PR with the changes!

2023-10-20

Maximilian Mattessich (15:36:11): > This time our problem has migrated to the ARM64 builder on macOS 13.1 (kjohnson1,link). Is there any significance to this test being shown separately to the other tests on the‘All Results’ page? > > We are somewhat confused why this error (Pandoc error 9) would occur only on this specific builder. Has anyone had similar build errors with pandoc? > > Even more confusingly, we appear to have passed all tests with this same builder (kjohnson1) on October 10th, and have not pushed any updates since then. Somehow this error was introduced in the October 16th build? > > * checking for file 'nipalsMCIA/DESCRIPTION' ... OK > * preparing 'nipalsMCIA': > * checking DESCRIPTION meta-information ... OK > * installing the package to build vignettes > * creating vignettes ... ERROR > --- re-building 'Vignette1.Analysis-of-MCIA-Decomposition.Rmd' using rmarkdown > --- finished re-building 'Vignette1.Analysis-of-MCIA-Decomposition.Rmd' > > --- re-building 'Vignette2.Single-Cell-Analysis.Rmd' using rmarkdown > Error: processing vignette 'Vignette2.Single-Cell-Analysis.Rmd' failed with diagnostics: > pandoc document conversion failed with error 9 > --- failed re-building 'Vignette2.Single-Cell-Analysis.Rmd' > > --- re-building 'Vignette3.Predicting-New-Scores.Rmd' using rmarkdown > --- finished re-building 'Vignette3.Predicting-New-Scores.Rmd' > > SUMMARY: processing the following file failed: > 'Vignette2.Single-Cell-Analysis.Rmd' > > Error: Vignette re-building failed. > Execution halted >

Andres Wokaty (16:05:25) (in thread): > The reason that the macos arm64 builds are shown separately is because their cadence is different than the other builders, which in general run daily whereas the macos arm64 builders are weaker and run weekly. > > The version of pandoc is different on the macos arm64 builds in that is newer. The other build machines generally have an older version of pandoc due to incompatibilties. > > It’s not exactly clear what is the issue. Pandoc wasn’t modified on that machine during this period of time; however, we do know that there were network issues related to the hubs. Is it possible that your package might have been impacted by this?

Maximilian Mattessich (16:10:43) (in thread): > @Andres Wokatywe were thinking that the issue might be related to the size of the assets loaded in by our Vignette 2. If these network issues occurred while loading the data, maybe this could have caused the error? Admittedly this seems pretty improbable. > > Do you know if there will be a re-build before the 3.18 submission deadline?

Andres Wokaty (16:12:22) (in thread): > The next macos arm64 3.18 build produce a report on Monday, which will be the last before the release.

Maximilian Mattessich (16:22:34) (in thread): > @Andres WokatyIt’s a little tough to find information on pandoc error 9, but alone github issueseems to indicate it’s a memory problem. Do the builders have different memory configurations that could be causing us to fail on kjohnson1 specifically? - Attachment: #1412 pandoc document conversion failed with error 9 > Hello,
> I am getting the following error when knitting .Rmd file to .html output. Can you please help me fix this issue or let me know what “Error 9” means? > > > pandoc document conversion failed with error 9 >

Andres Wokaty (16:30:21) (in thread): > the builders do have different memory sizes but the memory isn’t something that would have changed during the week. i saw that issue but it’s not very helpful:slightly_smiling_face:

Maximilian Mattessich (16:34:00) (in thread): > Ah ok, we will continue looking for the source of the error. Thanks so much for your help!

Andres Wokaty (17:00:34) (in thread): > I wonder if you have an intermittent problem because I ranR CMD buildmanually on kjohnson1: > > kjohnson1:tmp biocbuild$ R CMD build nipalsMCIA > * checking for file 'nipalsMCIA/DESCRIPTION' ... OK > * preparing 'nipalsMCIA': > * checking DESCRIPTION meta-information ... OK > * installing the package to build vignettes > * creating vignettes ... > OK > * checking for LF line-endings in source and make files and shell scripts > * checking for empty or unneeded directories > * looking to see if a 'data/datalist' file should be added > * building 'nipalsMCIA_0.99.7.tar.gz' > > Does your package need to access a resource via the internet?

Maximilian Mattessich (17:13:23) (in thread): > We do load a large (672 MB) dataset from our latest github release to build Vignette 2.

Maximilian Mattessich (17:14:17) (in thread): > I guess the network issues may have interrupted that process?

Andres Wokaty (17:18:53) (in thread): > That might be possible. We have some guidelines about querying resources athttps://contributions.bioconductor.org/querying-web-resources.html. I’m going to loop in@Hervé Pagèsand@Marcel Ramos Pérezwho might have additional recommendations about best practices but they might be offline already for the day.

Maximilian Mattessich (18:01:18) (in thread): > We are taking a look! Hopefully we can avoid these errors in the future.

Hervé Pagès (19:18:06) (in thread): > @Maximilian MattessichBefore we worry about the intermittent pandoc errors, I’d like to focus your attention on a couple of important issues with the “Single Cell Analysis” vignette (Vignette2.Single-Cell-Analysis.Rmd). Thedata-all-downloadcode chunk downloads 729M of experimental data from GitHub, without caching it. Bioconductor packages should not download data from GitHub (seehttps://contributions.bioconductor.org/data.html#other-data). Instead we ask that the data be put on ExperimentHub, and accessed via a dedicated data experiment package. If, for whatever reason, the ExperimentHub approach is discarded, then at the very least the data should be cached, typically usingBiocFileCache, so it can be reused across sessions without the need to redownload it each time. > Many code chunks are not evaluated (eval = FALSE) and the output of the commands have been copied and pasted in the Rmd document. This is not good practice and we strongly recommend against it as there’s no guarantee that the static output will remain in sync with the real output in the future (and experience has shown that it will inevitably start to diverge, probably sooner than you would think). If the reason for using these static code chunks is to make sure that the 3 vignettes can build in less than 5 min, then please consider reducing the size of the data used in the vignettes, or consider moving (some of) your vignettes to a separate workflow package. Then keep a minimalistic vignette innipalsMCIA. > Also please use the package canonical URL (a.k.a. package short URL) in your documentation e.g.https://bioconductor.org/packages/TENxPBMCDatainstead ofhttps://bioconductor.org/packages/3.15/data/experiment/html/TENxPBMCData.htmlThese recommendations apply to all the vignettes in the package so please make sure to bring the 3 vignettes in conformance with our guidelines. Thanks for your contribution to Bioconductor.

2023-10-23

Alan Murphy (09:54:17): > Hey, has the CPU/RAM allowance changed on the linux machines used in the bioc builds? I had previously just been running unit tests for MungeSumstats on linux 64 bit machines but the nightly check seems to now be killed on nebbiolo2:https://bioconductor.org/checkResults/3.18/bioc-LATEST/MungeSumstats/nebbiolo2-checksrc.html. It is passing on the other linux machine kunpeng2, is there much of a difference between these?

Anna Konstorum (12:36:56) (in thread): > @Hervé Pagès, popping into the conversation as a package co-developer along with@Maximilian Mattessich; I want to say we really appreciate your feedback. After consideration, we do believe that in the long run, moving Vignette 2 to a Workflow package will be optimal, as it will allow us to run all code blocks with eval=TRUE and elaborate on the details more fully, while keeping a slimmed down version in thenipalsMCIApackage. > > We will note that our current decisions for this vignette were based on maintaining a balance between vignette size and package dependency with providing enough detailed information for users interested in single-cell analysis to usenipalsMCIA. Our reasoning for not usingBiocFileCachewas that we would prefer to leave the decision to the users whether to have a cache or not. UsingBiocFileCachewould add an extra layer of complexity in addition topiggyback, which downloads files to a temporary directory (which can be made permanent if the user wishes) directly without have to interact with a Cache object, and syncs with each release. We also did not feel it appropriate in our case to create a dedicated data package since the data we use in the vignette is not our data, and there are already several existing Bioconductor packages that make public single cell data (including ours) available. The data that we have on GitHub, and is uploaded viapiggyback, are mainly Seurat objects that are created following data processing in the eval=FALSE chunks. We did not expect this data to change significantly, and did not want to require additional dependecies that would only be relevant for this vignette, hence why we maintained the chunks as eval=FALSE and decided to upload their outputs. But, we do appreciate your points with respect to our decisions here. > > We certainly do feel that your suggestion re Workflow package + minimal vignette is optimal, and where we should go with this vignette, but we did want to elaborate on our choices for the vignette in its current state. > > We also want to note that the other vignettes do not have these challenges. > > In light of this discussion, we would like to ask for your advice with regards to these planned updates. Given that the deadline for updates to all packages was Oct. 20th in preparation for the release, we do not feel it would wise to try and implement such extensive changes in the next few days. We would appreciate if our package was given the opportunity to be included in the release, and so defer to your judgment on how to proceed.

Anna Konstorum (12:46:49) (in thread): > For additional context on our communication with the BioConductor team on this vignette, please see our response to@Laurent GattoGatto’s review under ‘Package Data’ (part 2)here, as well as his response (again, under ‘Package Data’)here. I hope this can help clarify what we have previously considered, and we look forward to continuing to optimize our package in consideration of the BioConductor guidelines and best practices.

Michael Lawrence (14:51:54): > I’m curious why HelloRanges is failing only on Linux, due to a missing hg19.genome file, which is supposed to come from HelloRangesData and should be found in the working directory set by the vignette. Any help would be appreciated.

Martin Grigorov (14:58:29) (in thread): > on kunpengg2: > > > BiocManager.install("HelloRangesData") > Error in BiocManager.install("HelloRangesData") : > could not find function "BiocManager.install" > > BiocManager::install("HelloRangesData") > Bioconductor version 3.18 (BiocManager 1.30.22), R 4.3.1 (2023-06-16) > Old packages: 'agricolae', 'flextable', 'RPostgres', 'spam', > 'spatstat.explore', 'spatstat.model', 'utf8' > Update all/some/none? [a/s/n]: n > Warning message: > package(s) not installed when version(s) same as or greater than current; use > `force = TRUE` to re-install: 'HelloRangesData' > > BiocManager::install("HelloRangesData", force = TRUE) > Bioconductor version 3.18 (BiocManager 1.30.22), R 4.3.1 (2023-06-16) > Installing package(s) 'HelloRangesData' > trying URL '[https://bioconductor.org/packages/3.18/data/experiment/src/contrib/HelloRangesData_1.27.0.tar.gz](https://bioconductor.org/packages/3.18/data/experiment/src/contrib/HelloRangesData_1.27.0.tar.gz)' > Content type 'application/x-gzip' length 47955415 bytes (45.7 MB) > ================================================== > downloaded 45.7 MB > > * installing **source** package 'HelloRangesData' ... > **** using staged installation > **** inst > **** help > No man pages found in package 'HelloRangesData' > ***** installing help indices > **** building package indices > **** installing vignettes > **** testing if installed package can be loaded from temporary location > **** testing if installed package can be loaded from final location > **** testing if installed package keeps a record of temporary installation path > * DONE (HelloRangesData) > > The downloaded source packages are in > '/home/biocbuild/tmp/Rtmp4tro8M/downloaded_packages' >

Martin Grigorov (14:58:58) (in thread): > do you want me to try some function ?

Hervé Pagès (15:28:52) (in thread): > Thanks for your answers and for considering moving the vignettes to a dedicated workflow package. This sounds like a reasonable goal for BioC 3.19. > The repeated downloads are unfortunate. This means thatR CMD buildandR CMD checkwill download 729M of data each time they run. Given that these are the basic commands for performing QA on the package, expect them to be run frequently. For example our daily builds will run each command every night on every build machines, in release and devel. This means that the 729M of data will be downloaded 18 times each night. This is why we require that the data be cached. It’s ok if you want to make the caching optional for your users, but caching should be the default. > Thanks again for your contribution to Bioconductor.

Martin Grigorov (15:32:29) (in thread): > > R CMD build HelloRanges > * checking for file 'HelloRanges/DESCRIPTION' ... OK > * preparing 'HelloRanges': > * checking DESCRIPTION meta-information ... OK > * installing the package to build vignettes > * creating vignettes ... OK > * checking for LF line-endings in source and make files and shell scripts > * checking for empty or unneeded directories > * building 'HelloRanges_1.27.0.tar.gz' >

Michael Lawrence (16:14:27) (in thread): > Thanks for your help. So it should build successfully next time?

Hervé Pagès (16:45:09) (in thread): > Weirdly,HelloRangesData/extdata/is empty on nebbiolo2: > > biocbuild@nebbiolo2:~/bbs-3.18-bioc/R/site-library/HelloRangesData$ tree > . > ├── DESCRIPTION > ├── extdata > ├── help > │ ├── aliases.rds > │ ├── AnIndex > │ ├── HelloRangesData.rdb > │ ├── HelloRangesData.rdx > │ └── paths.rds > ├── html > │ ├── 00Index.html > │ └── R.css > ├── Meta > │ ├── features.rds > │ ├── hsearch.rds > │ ├── links.rds > │ ├── package.rds > │ └── Rd.rds > └── NAMESPACE > > 4 directories, 14 files > > I’m taking a closer look now, trying to understand how this could happen.

Hervé Pagès (17:03:53) (in thread): > Not sure what’s going on. I’m starting to suspect that “someone” (e.g. a vignette, example, or unit test somewhere) is deleting the content of the folder but I was not able to identify the culprit yet. There’s still the question of why we only see this on Linux. More detective work will be required. In the mean time I’ve reinstalledHelloRangesDataon nebbiolo2.

Martin Grigorov (17:28:54) (in thread): > R CMD check ...still fails on kunpeng2:confused:Andextdata/is empty too

Martin Grigorov (17:31:57) (in thread): > re-installingHelloRangesDatapopulatesextdata/!

Martin Grigorov (17:39:41) (in thread): > R CMD check HelloRanges_1.27.0.tar.gzdeletes it

Martin Grigorov (17:51:25) (in thread): > > Loading required package: GenomicAlignments > [1] TRUE > > HelloRanges:::.test() > Error in library("RUnit", quietly = TRUE) : > there is no package called 'RUnit' > Calls: <Anonymous> -> <Anonymous> -> library > Execution halted > * checking for unstated dependencies in vignettes ... NOTE > 'library' or 'require' call not declared from: 'TxDb.Hsapiens.UCSC.hg19.knownGene' > * checking package vignettes in 'inst/doc' ... OK > > RUnit needs to be added toSuggestsnot sure what should be done aboutcall not declared

Michael Lawrence (18:10:31) (in thread): > Thanks, I added RUnit and TxDb.Hsapiens.UCSC.hg19.knownGene to Suggests. But I have no idea why checking HelloRanges would delete that directory (and only on Linux).

Michael Lawrence (18:12:33) (in thread): > This is a stable package that has been building without issue for years, so something weird is just now happening on that system.

Anna Konstorum (18:26:28) (in thread): > @Hervé Pagès, thank you for these insights. Given the burden that the lack of a default cache is creating for the build machines, we will prioritize modifications to a default cache for the next version bump.

Hervé Pagès (18:56:11) (in thread): > Not just now.HelloRangeshas not buit once in 3.18 (there’s still no source tarball available), and was also failing in 3.17 for a while. Maybe it has something to do with the use ofsite-libraryto install non-base packages. We started to use that setup in Spring and on Linux only.

Michael Lawrence (19:14:30) (in thread): > Well in that case perhaps it can be deprecated, but this might be a deeper issue that would be good to identify.

Hervé Pagès (19:52:55) (in thread): > > R CMD check HelloRanges_1.27.0.tar.gz deletes it > Are you sure about that@Martin Grigorov? I’m not able to reproduce on nebbiolo2 or on my laptop.

2023-10-24

Martin Grigorov (02:18:40) (in thread): > let me try it one more time with the updated dependencies/suggests

Martin Grigorov (02:50:40) (in thread): > the build and check passed successfully on kunpeng2 !

koki (21:13:08): > Dear, BioC Core team@Vince Carey@Kayla Interdonato@Lori ShepherdI’m a developer ofiTensorwhich is a CRAN package and my package has the dependency ofmixOmicspackage. > ThemixOmicspackage has now CHECK error on Linux machine of BioC server.https://bioconductor.org/checkResults/3.18/bioc-LATEST/mixOmics/nebbiolo2-checksrc.htmlThe error seems that it is because ofdevtoolsis not available. > Is this error a problem on themixOmicsside and should be handled by the developer? > Or is it just a temporary unavailability ofdevtoolson the BioC server and therefore we should just wait it out?

Hervé Pagès (21:25:16) (in thread): > devtoolsis installed on the Linux builders. However,mixOmicsdoes not listdevtoolsin itsSuggestsfield (despite using it in their vignette) henceR CMD checkcomplains that it’s not available (thisR CMD checkbehavior is controlled by the*R_CHECK_SUGGESTS_ONLY*=truesetting). > It’s the responsibility of themixOmicsdevelopers/maintainers to fix this. > You can try to ping them by opening an issue herehttps://github.com/mixOmicsTeam/mixOmics/issuesor by contacting them by email directly (don’t hesitate to include all the authors for which an email address is provided in the DESCRIPTION file in addition to the official maintainer). I believe some BioC core members have tried this already but with no positive results so far.

Lori Shepherd (21:44:53) (in thread): > Mixomics responded within the last 24 hours they pushed up a fix . It hopefully should be available in the next day

2023-10-25

koki (00:16:00) (in thread): > @Lori Shepherd@Hervé PagèsThank you for your prompt response.

Lori Shepherd (16:13:14): > Bioconductor Core Team is pleased to release Bioc 3.18! Thank you to all developers and community members for contributing to the project. The full release announcement can be found at:https://bioconductor.org/news/bioc_3_18_release/

2023-10-27

Jacques SERIZAY (03:15:39): > Congratulations and thanks to the core team for releasing 3.18:tada:I’m now having a Cairo related issue, only on merida1 (seehttps://bioconductor.org/checkResults/devel/bioc-LATEST/HiCExperiment/merida1-buildsrc.html): > > Cairo-based devices are not available for this platform > > Is it something expected? Thanks!

Lori Shepherd (07:12:45) (in thread): > Thank you for reaching out. We noticed this too and are currently investigating.

Sagun Maharjan (12:17:33): > Hi all, > Thank you so much for the suggestions on build errros last time. It was very helpful. For the following Macarron pacakage, we are now seeing theIconwaybuild error for Mac. Any suggestions please?https://bioconductor.org/checkResults/release/bioc-LATEST/Macarron/lconway-buildsrc.html

2023-10-28

sahitya (09:24:10): > @sahitya has joined the channel

2023-10-31

Dima Lvovs (11:15:44): > Hello, are thebuild RSSfeeds supposed to return the daily build status? Trying to subscribe for package CoGAPS usinghttp://bioconductor.org/rss/build/packages/CoGAPS.rss, and getting<updated>2022-04-28T14:32:27-04:00</updated>

Lori Shepherd (11:23:41) (in thread): > Thanks for the reminder; I meant to look into package level RSS feeds when a bug was reported before. Sorry for the inconvenience but we’ll investigate

2023-11-02

Lori Shepherd (14:33:46) (in thread): > I think I found the bug and pushed up a fix that hopefully will have better individual package updating moving forward. Per your question: the rss feed for a package should only change when there is a change is one of the phases of the build report (install, build, check) so its not completely reflecting a daily report but will reflect any changes with the report (at least that is my understanding)

2023-11-03

Dima Lvovs (10:50:06) (in thread): > great news, thank you! I’ll try to subscribe to some more apps rss and will let you know when it worked.

2023-11-06

Andres Wokaty (16:58:24) (in thread): > Hi@Sagun MaharjanSorry it took time to follow up. > > It appears that the build products you’re saving in/vignettesare causing some conflict on the Macs: > > lconway:vignettes biocbuild$ ls > MACARRoN.Rmd Macarron.html Macarron_output > > I cloned Macarron on lconway and was able toR CMD buildafter removingMacarron.htmlandMacarron_output. We recommendremoving build productssince they’ll be regenerated and when they’re regenerated, you should create them in atemporary directory. It also might be better if the vignette capitalization matched that of the package.

Sagun Maharjan (16:59:08) (in thread): > Hi@Andres Wokaty, > Thank you so much for the detailed message. We will try this.

2023-11-13

Alan O’C (11:48:10): > Note from Matrix devs relating to non-CRAN repos:https://stat.ethz.ch/pipermail/r-package-devel/2023q4/010051.html

Hervé Pagès (12:04:18) (in thread): > Thanks@Alan O’C. They actually only listed CRAN packages:thinking_face:Only Bioconductor package that is a reverse LinkingTo isbcSeqand they omitted it. We’ll ask thebsSeqfolks to bump the version.

Alan O’C (12:06:45) (in thread): > I think eg irlba needs to be rebuilt on the build machines with the new Matrix version, at least I’m seeing thisas_cholmod_sparseerror in the build reports (https://master.bioconductor.org/checkResults/3.18/bioc-LATEST/scater/nebbiolo2-buildsrc.html) that is apparently fixed by reinstalling irlba after updating matrix (https://github.com/bwlewis/irlba/issues/70)

Hervé Pagès (12:15:09) (in thread): > Wait, what? They know that the binaries of these rev deps that are already in the wild are incompatible and they’re not going to fix that? (by asking their maintainers to bump the versions so new binaries are made and propagated). Instead they expect everybody to reinstall these packages from source? Completely unrealistic.

Alan O’C (12:15:34) (in thread): > :shrug:

Alan O’C (12:17:33) (in thread): > I could email those maintainers asking them to bump versions, and CC the matrix maintainer if you want to get a bit passive-aggressive:slightly_smiling_face:

Hervé Pagès (12:18:37) (in thread): > That sounds like an excellent strategy:laughing:Thanks for volunteering!

Alan O’C (13:14:38) (in thread): > I did BCC, at least

Lori Shepherd (13:20:18): > There is an announcement athttps://cran.r-project.org/ > > Dear users, > > The CRAN Admin Team will perform system upgrades during the period Wednesday November 15 until Thursday November 16, 2023. > There will be intermittent outages in service during this time. > We thank you for your understanding and apologise for any inconvenience this may cause. > > CRAN Admin Team > > This may effect BiocManager and the ability to download/install packages. > We appreciate your patience and understanding during the CRAN update period.

Lukas Weber (14:52:10): > Hi, a couple of my packages (nnSVG,ggspavis) are failing in checks when building vignettes on the builders, during steps where data objects are loaded from ExperimentHub inSpatialExperimentformat. The following ExperimentHub objects are affected right now: EH7538, EH7539. These are both loaded from theSTexampleDatapackage. Is there a way to check if something is wrong with these objects on the builders?

Lori Shepherd (14:55:48) (in thread): > yes I can check. both release and devel?

Lukas Weber (14:56:49) (in thread): > Great, thank you. Yes, both release and devel are affected

Lukas Weber (15:00:05) (in thread): > I just checked again – also object EH7540 (in devel only) is affected. It seems to be several / all of the objects in theSTexampleDatapackage

2023-11-15

Hervé Pagès (14:04:40) (in thread): > Thx. About 40+ Bioconductor software packages are broken with the “function ‘as_cholmod_sparse’ not provided by package ‘Matrix’” error.irlbaand all CRAN packages got automatically reinstalled on the Linux builders after we reinstalled/updated R on these machines yesterday, but we still see the error on the Windows, Mac, and Linux arm64 builders. > Unfortunately reinstallingirlbaon our build machines doesn’t solve anything for the end user: the 40+ Bioconductor packages will remain broken for them until they reinstallirlba(plus who knows what other CRAN package) on their machine. > Hopefully all the CRAN packages that need reinstallation will realize the importance of bumping their versions ASAP!:crossed_fingers:

Xiaosai Yao (22:42:03): > @Xiaosai Yao has joined the channel

2023-11-16

Marc Elosua (15:41:40): > @Marc Elosua has joined the channel

2023-11-17

Marc Elosua (09:35:02) (in thread): > Hi! Thank you so much for looking into this and addressing it! Wanted to document here that I’m getting a relatedMatrix > > ══ Failed tests ════════════════════════════════════════════════════════════════ > ── Error ('test-SPOTlight.R:12:1'): (code run outside of `test_that()`) ──────── > Error in `validObject(.Object)`: invalid class "LogMap" object: superclass "mMatrix" not defined in the environment of the object's class > > Which has been reportedhere&hereas aMatrixrelated error > > Somehow the build works for Linux but returns an error in Windows and macOS. > > Any thoughts on how I should go about this? - Attachment: #8021 Error in CreateSeuratObject - Attachment: Comment on #7993 FindNeighbors Error

Alan O’C (09:37:51) (in thread): > I’ve no idea personally, sorry

Hervé Pagès (11:01:22) (in thread): > Same story: the latestMatrixis not backward binary-compatible with some already installed packages. In this case it breaks theLogMapclass fromSeuratObject: > > > library(SeuratObject) > > LogMap(y=letters) > Error in validObject(.Object) : > invalid class "LogMap" object: superclass "mMatrix" not defined in the environment of the object's class > > Won’t go away until theSeuratObjectfolks bump the package version.

Hervé Pagès (11:05:15) (in thread): > This breaks about 22 Bioconductor packages.

Alan O’C (11:07:10) (in thread): > My experience reporting stuff to Seurat folks is to not hold your breath for a quick fix

Marc Elosua (11:07:41) (in thread): > Not sure its best practice or if it makes sense but I’m trying to specify Matrix version in the DESCRIPTION > > Imports: > ggplot2, > NMF, > Matrix (== 1.6-1.1), > matrixStats, > nnls, > SingleCellExperiment, > stats >

Alan O’C (11:11:11) (in thread): > I think exact version specs are at best heavily discouraged

Alan O’C (11:12:05) (in thread): > Also, this would mean your package is still broken on the build machines, that use the newer matrix version… fun times

Amanda Hiser (14:02:40): > Hello! I’m trying to check on the devel build status of an annotation package, but it looks like there isn’t a devel build page for the annotation packages. Am I misinterpreting that? Here’s the page for the ann packages:https://bioconductor.org/checkResults/devel/data-annotation-LATEST/, as well as experiment (for comparison):https://bioconductor.org/checkResults/devel/data-experiment-LATEST/

Lori Shepherd (14:05:23): > we branched the annotations today. build reports for the annotations should resume next week

Amanda Hiser (14:22:29): > Awesome, thanks!

Hervé Pagès (14:27:09) (in thread): > > My experience reporting stuff to Seurat folks is to not hold your breath for a quick fix > https://github.com/mojaveazure/seurat-object/issues/172I won’t hold my breath then:wink:

Marc Elosua (14:29:49) (in thread): > Thank you!!!

2023-11-18

Dima Lvovs (22:11:03) (in thread): > hurray, I got a status update into the RSS reader, thanks!

2023-11-22

Alan Murphy (10:58:19): > Hey! My package is failing on nebbiolo2 (linux) on Bioconductor 3.18 release as a package isn’t installed it needs: SNPlocs.Hsapiens.dbSNP155.GRCh38. Can this be added?

Lori Shepherd (11:02:33): > what is the name of your package? Did you declare it in your DESCRIPTION of your package as a dependency?

Alan Murphy (11:38:22) (in thread): > The package is MungeSumstats - these are large database packages that I was advised to include only in the Suggests to help with package size

Lori Shepherd (11:39:49) (in thread): > yes just wanted to make sure it was in Suggests that is correct

Lori Shepherd (11:41:03) (in thread): > It looks like it passed todayhttps://bioconductor.org/checkResults/3.18/bioc-LATEST/MungeSumstats/

Alan Murphy (11:46:05) (in thread): > Ah apologies, I should have waited another half hour!

Farhan Ameen (21:43:40): > @Farhan Ameen has joined the channel

2023-12-05

Asier Ortega (03:17:11): > @Asier Ortega has joined the channel

2023-12-06

Henrik Bengtsson (21:08:08): > Hello, could some of you please confirm or dispute that you’re also getting: > > > example("read_somascan", package = "autonomics") > rd_sms> file <- download_data('atkin18.somascan.adat') > rd_sms> read_somascan(file, pca = TRUE, fit = 'limma', block = 'Subject_ID') > Error in (1 + f_col):n_col : NA/NaN argument > > with autonomics 1.10.2? I’m getting this error on multiple systems, but the Bioc checks don’t pick it up, cf.https://support.bioconductor.org/p/9155535/.

2023-12-07

Peter Hickey (00:03:04) (in thread): > can confirm with R 4.3.1 withaarch64-apple-darwin20usingv1.10.2ofautonomicsand up-to-date BioC installation (BiocManager::valid()isTRUE)

Henrik Bengtsson (01:13:38) (in thread): > Thank you for confirming. I’ve tested on various Linux systems with R 4.3.2. So, this makes me question the Bioc check results here, cf.https://bioconductor.org/checkResults/release/bioc-LATEST/autonomics/.

Martin Grigorov (04:18:09) (in thread): > https://bioconductor.org/checkResults/devel/bioc-LATEST/autonomics/kunpeng2-buildsrc.htmlshows the error too for v1.11.5 !

Vince Carey (11:58:07) (in thread): > i can confirm too. will check further after TAB call …

Marcel Ramos Pérez (12:19:16) (in thread): > I can reproduce the error on theRELEASE_3_18container : > > Error in (1 + f_col):n_col : NA/NaN argument > > packageVersion("autonomics") > [1] '1.10.2' >

2023-12-08

Amanda Hiser (12:26:05) (in thread): > Hi@Henrik Bengtsson! This may not directly answer your question, but if you are unable to resolve your issue withautonomics, I suggest you check outSomaDataIO::read_adat()on CRAN. It’s developed (and actively maintained) by SomaLogic for reading SomaScan data into R

Henrik Bengtsson (13:36:14) (in thread): > Thank you@Amanda Hiser. I should have made it more clear here in Slack that my interest in this one is why the Bioconductor check servers do not report on this, and whether there could be something that needs to be fixed there? Inhttps://support.bioconductor.org/p/9155535/, I tracked it down to be due to corrupt online data being downloaded and that causes the error. But then, Bioconductor checks should see the same, soI think there’s something wrong with the Bioconductor check servers, e.g. data is pull from a local cache/proxy and not from the upstream source (which we do use above). Hopefully, it’s just this package, but in the worst case the underlying cause might affect other packages. I went down this rabbit whole because I got errors on my reverse-dependency checks ofmatrixStats, while Bioconductor saidautonomics is all good, which I no longer think is true.

2023-12-09

Vince Carey (10:13:08) (in thread): > Thanks Henrik. Definitely want to get to the bottom of this.@Hervé Pagèshas been traveling.

2023-12-11

Henrik Bengtsson (19:26:40) (in thread): > UPDATE: It looks like@Mike Smithtracked it down to a dirty file cache, cf.https://support.bioconductor.org/p/9155535/#9155621.

Henrik Bengtsson (19:35:03) (in thread): > Is there an issue tracker for the Bioconductor check system? Would that behttps://github.com/Bioconductor/BBS,https://github.com/Bioconductor/BiocCheck, or something else? (I’d like to propose that each package check will start off with a clean, temporary cache.)

Hervé Pagès (20:49:04) (in thread): > RunningR CMD checkon 2000+ packages without any caching would require downloading 250GB of data everyday on 8 build machines. That’s 2TB of data, daily! Not realistic. > In this case I would argue that the issue is withautonomics::download_data(). The function wants to use caching which is fine but it needs to be done properly i.e. it should be able to detect that what’s in the cache is stale and redownload it if needed. I don’t know what provisions are available inBiocFileCacheto deal with cache invalidation, I’m not familiar enough with the package, but it sounds like this is something it should be able to handle.@Lori Shepherd?

Hervé Pagès (20:55:30) (in thread): > @Andres Wokaty@Lori ShepherdCan we nukeautonomicscache on the builders? Note that this is not needed on nebbiolo1 or kunpeng2 whereR CMD checkseems to produce the expected error:https://bioconductor.org/checkResults/3.19/bioc-LATEST/autonomics/

2023-12-12

Mike Smith (03:21:08) (in thread): > I think the issue is that the siteautonomics::download_data()is grabbing the files from doesn’t include aetagorlast-modifiedheader in the HTTP response.BiocFileCachehas provisions for using those to detect stale caches and update viabfcneedsupdate()but I think the package author needs to come up with something bespoke if those headers are missing. > > I’ll tag@Aditya Bhagwatto highlight this discussion of their package.

Henrik Bengtsson (10:13:43) (in thread): > > RunningR CMD checkon 2000+ packages without any caching would require downloading 250GB of data everyday on 8 build machines. That’s 2TB of data, daily! Not realistic. > 1. Is this design decision documented somewhere? That can be useful to know as for the package maintainer, but also for others (e.g. those who run revdep checks and trying to understand discrepancies). > > 2. Definitely not an expert, but aren’t there HTTPS proxies that can help shield against this? E.g. Squid or NGINX? Maybe worth adding to the to-do list of things to look into? > > 3. I don’t how much one can generalize it, but for common locations, the check system could report on any content found in: > > path <- tools::R_user_dir("autonomics", which = "cache") > files <- list.files(path, recursive = TRUE, all.files = TRUE) > if (length(files) > 0) { > ... > } > > I can see howR CMD checkshould do this itself, but this could also be added to the Bioconductor check pages. It could even report of files before and after each check, and any diffs. > > 4. I’d love a way to control this whetherR CMD checkshould run in a 100% clean environment or not, e.g. checking with something likeR_BIOCFILECACHE_DIR="{session}"would causeBiocFileCacheto use a temporary, session-specific cache folder. > > FWIW, when I run revdep checks onmatrixStats,futureet al., … I almost always run into to false-positive check ERRORs or WARNINGs with Bioconductor packages (very very rare for CRAN packages) due to what I believe are due to results cached to file. Sincerevdepcheckchecks run current and new version of each package simultaneously, I think I also observe cache clashes, e.g. something is writing when the other check instance is reading. For some Bioconductor packages, triggering a rerun might solve it. In other cases, the best shot is to check those packages sequentially, i.e. a singleR CMD checkat the time. For the longest, I thoughautonomicswas one of those packages, but turns out there was another reason.

Hervé Pagès (12:04:41) (in thread): > I don’t think we can do much about it. Caching happens all the time everywhere in many different forms, not just at the R level but also at the system level, so it would be very hard to start with a clean plate each time we runR CMD checkcommand on a package. Most of the time this is not an issue since using cached data should be semantically equivalent to using the original data, except when it’s not like here, which is a rare situation (hopefully). We could mitigate this by nuking~/.cache/R/on the build machines on a regular basis e.g. once or twice a month. Would that be a good compromise?

2023-12-19

Luke Zappia (07:52:38): > Would it be possible to reset the****{basilisk} ****cache for****{zellkonverter}****devel? The builds are failing with a Python error that I can’t reproduce anywhere else (including release which is passing fine). I’m also open to other suggestions if anyone has another idea for how to debug this.

Martin Grigorov (07:58:44) (in thread): > Hi! I see it fails at BUILD for Linux ARM64:https://bioconductor.org/checkResults/3.19/bioc-LATEST/zellkonverter/kunpeng2-buildsrc.htmlWhich version of gcc/c++ compiler does it require ?

Martin Grigorov (08:02:20) (in thread): > currentlykunpeng2provides GCC/C++ 10.3.1 > and/usr/lib64/libstdc++.so.6provides symbols uptoGLIBCXX_3.4.28

Luke Zappia (09:15:25) (in thread): > I’m not sure TBH, that error happens when installing the environment so maybe****{basilisk}****controls that? It is using Python 3.10 so maybe that also makes a difference? > > I was mostly looking for help with error on the other systems which seems to be in one particular test rather than building the package.

Vince Carey (09:56:02): > This is primarily an internal (to core) message, but I post it here in case there are readers who have made progress on this – what would it take to produce an “R cache signature” so that two cache collections could be compared to identify differences? Hopeless? Assume we are just considering AnnotationHub, ExperimentHub, and the default BiocFileCache@Henrik Bengtsson?

Andres Wokaty (10:53:15) (in thread): > I’ll delete the caches.

Vince Carey (15:33:14) (in thread): > FWIW withdocker run --entrypoint /bin/bash -tighcr.io/bioconductor/bioconductor_salt:22.04-bioc-3.19-r-developmentI built and checked zellkonverter without error. It would be nice to know what aspect of the cache is problematic ….

Spencer Nystrom (16:04:14) (in thread): > What would you consider “identifying differences”? How granular would you like this to be to consider it successful?

Kasper D. Hansen (19:22:46) (in thread): > Seems like md5sum would be the place to start

2023-12-20

Luke Zappia (02:04:42) (in thread): > I’m not actually sure. I haven’t been able to reproduce the error either so my guess was that it might have something to do with the cached environment on the build systems.

2023-12-22

Hervé Pagès (13:13:39) (in thread): > Are you thinking of something similar tosessionInfo()but that describes the state of the R cache@Vince Carey?

Vince Carey (14:05:51) (in thread): > That’s right Hervé … in fact I think it would be a little more involved, because we don’t have a comparison operation for sessionInfo(), do we? But our aim would be to publish the signature of the cache on the BBS so that developers (and users?) would be able to verify that their cache(s) are compatible with those of the BBS.

2023-12-28

David Rach (08:59:58): > @David Rach has joined the channel

2024-01-05

Dima Lvovs (12:14:58): > Hello, are there specific requirements for the BiocFileCache location to enable daily builds use the cache instead of downloading the data for vignettes every day? I’ve currently placed it in the user directorypath = tools::R_user_dir(package = "packagename")for one package, and wanted to confirm it’s OK before doing same for a couple of more packages.

Hervé Pagès (15:09:44) (in thread): > tools::R_user_dir(package="packagename", which="cache")is what you want. Note that this is what theBiocFileCache()constructor uses by default so you actually don’t need to explicitly set this.

Hervé Pagès (15:14:50) (in thread): > Small correction: by defaultBiocFileCache()usestools::R_user_dir(package="BiocFileCache", which="cache"), my bad. I guess it’s ok to just use that though, no need to use your own dedicated location.

Dima Lvovs (15:28:12) (in thread): > Thank you!

2024-01-17

Dima Lvovs (10:18:01) (in thread): > wouldn’t that mean that all packages will use the same cache location, and there could be name collisions?

Hervé Pagès (13:03:20) (in thread): > @Lori Shepherdcan provide more details but I believe that file names in the cache are mangled with unique ids to avoid that.

Lori Shepherd (13:07:45) (in thread): > correct. there is a way to override it but by default it adds a unique identifier to the file name that is downloaded to avoid conflict

2024-01-23

Alan O’C (11:04:00): > Got an error forscRNAseq::ZeiselBrainData()on merida1 devel (https://bioconductor.org/checkResults/devel/bioc-LATEST/scater/merida1-buildsrc.html): > > Corrupt Cache: index file > See AnnotationHub's TroubleshootingTheHubs vignette section on corrupt cache > cache: /Users/biocbuild/Library/Caches/org.R-project.R/R/ExperimentHub > filename: experimenthub.index.rds > > Can we nuke the cache? Thanks

Lori Shepherd (11:21:06): > we reset the cache’s yesterday; lets see if it resets on the next build

2024-01-31

Clemens Kohl (07:20:55): > @Clemens Kohl has joined the channel

2024-02-01

Kasper D. Hansen (14:38:30): > Continuing a conversation with@Andres Wokatyabout the macOS builds on apple silicon. I wiped all packages and resinstalled stuff from bioc-devel. My goal is to emulate someone who does not have build capability apart from (say) experiment data packages, so I answer “no” to any question about “installing newer packages from source” (I am fine with the fact that - due to build delays - a package may be newer in source form.

Kasper D. Hansen (14:38:52): > Observations, when I runBiocManager::install("minfi", dependencies = TRUE)

Kasper D. Hansen (14:40:39): > 3 packages have no binaries available:rhdf5,rhdf5filtersandHDF5ArrayThis now becomes a problem for all downstream dependencies. It seems weird that we switch from release (where the binary exists) to devel and we do not have a working binary for version X.Y.0

Kasper D. Hansen (14:41:08): > Furthermore - and this looks like a book - I am being told by the installation that

Kasper D. Hansen (14:41:12): > > There are binary versions available but the source versions are later: > binary source needs_compilation > annotate 1.81.0 1.81.2 FALSE > GenomicRanges 1.55.1 1.55.2 TRUE > SummarizedExperiment 1.33.2 1.33.3 FALSE > Biostrings 2.71.1 2.71.2 TRUE > > Do you want to install from sources the packages which need compilation? (Yes/no/cancel) no > Packages which are only available in source form, and may need > compilation of C/C++/Fortran: 'rhdf5' 'rhdf5filters' 'HDF5Array' > Do you want to attempt to install these from sources? (Yes/no/cancel) no >

Kasper D. Hansen (14:42:55): > Despite saying no here, later on in the process, it actually tries to install SummarizedExperiment 1.33.3 from source (see attached log) . That’s weird. In fact the error cascade I see in the installation seems weird to me, why isGenomeInfoDbDatamissing?

Kasper D. Hansen (14:43:16): - File (Plain Text): notes_devel_M2.txt

Marcel Ramos Pérez (16:22:30) (in thread): > I wonder if this can be disabled by settingoptions(pkgType = 'binary')perhaps it’s set to'both'

Kasper D. Hansen (16:30:54) (in thread): > Might be.Just weird that I say no and it does it anyway.I’llcheck when I have a chance

Hervé Pagès (19:54:30) (in thread): > This looks like a long standing bug ininstall.packages()where it does not necessarily install packages in the right order when the packages to install are a mix of sources and binaries that are spread across more than one repo.

Hervé Pagès (20:12:02) (in thread): > kjohnson3 is a new machine (Mac Pro Silicon, purchased at the end of 2023). > As you can see based on the nb of TIMEOUT’s here:https://bioconductor.org/checkResults/3.19/bioc-mac-arm64-LATEST/, it’s a very slow builder! > Compare to the Mac ARM64 report in release (kjohnson1, a Mac Studio):https://bioconductor.org/checkResults/3.18/bioc-mac-arm64-LATEST/If you compare the BUILD/CHECK times between the two machines, you’ll see that kjohnson3 is between 2 and 3 times slower than kjohnson1. For exampleR CMD build csawtakes 338.1 s on the former vs 107.6 s on the latter. > We’re still trying to figure out what’s going on. Needless to say that this new machine has been a big disappointment so far:disappointed:

Kasper D. Hansen (20:42:06) (in thread): > What’sthe model names?

Kasper D. Hansen (20:48:41) (in thread): > system_profiler SPHardwareDataType

Kasper D. Hansen (20:49:00) (in thread): > (in Terminal)

Kasper D. Hansen (20:49:41) (in thread): > I don’t think it is the order. Why does it go for the source packages when I say no? It seems to want to suck up the source packages

Hervé Pagès (20:51:01) (in thread): > We don’t produce binaries forGenomeInfoDbDataor for data packages in general.

Hervé Pagès (20:53:00) (in thread): > SoGenomeInfoDbDataneeds to be installed from source, this is not an option. Problem is thatinstall.packages()tries to install other packages that depend on it (e.g.GenomicFeatures)beforeinstallingGenomeInfoDbData.

Kasper D. Hansen (20:54:33) (in thread): > Should we do binaries for those packages? Would require more space but would it make things easier?

Kasper D. Hansen (20:55:02) (in thread): > Yes, on my system it is set to"both"But I think that might be the default

Hervé Pagès (20:55:43) (in thread): > Why? These are pure R packages so binaries are not needed. Better to report the bug ininstall.packages().

Kasper D. Hansen (20:57:10) (in thread): > What happens if you set yourpkgTypeto only get binaries?

Hervé Pagès (20:58:33) (in thread): > > kjohnson3:~ biocbuild$ system_profiler SPHardwareDataType > Hardware: > > Hardware Overview: > > Model Name: Mac Pro > Model Identifier: Mac14,8 > Enclosure: Rack > Model Number: Z172000TRLL/A > Chip: Apple M2 Ultra > Total Number of Cores: 24 (16 performance and 8 efficiency) > Memory: 192 GB > System Firmware Version: 10151.41.12 > OS Loader Version: 8422.141.2.700.1 > Serial Number (system): G4NL9W9XNC > Hardware UUID: 360759C8-A272-58A9-97B9-B9F743AE7DB0 > Provisioning UDID: 00006022-001829260E90A01E > Activation Lock Status: Disabled >

Hervé Pagès (21:06:59) (in thread): > You probably won’t be able to install any data package if you do that.

Kasper D. Hansen (21:17:12) (in thread): > what does kjohnson1 look like?

Hervé Pagès (21:38:33) (in thread): > > kjohnson1:~ biocbuild$ system_profiler SPHardwareDataType > Hardware: > > Hardware Overview: > > Model Name: Mac Studio > Model Identifier: Mac13,1 > Model Number: Z14J000H4LL/A > Chip: Apple M1 Max > Total Number of Cores: 10 (8 performance and 2 efficiency) > Memory: 64 GB > System Firmware Version: 10151.41.12 > OS Loader Version: 8422.141.2.700.1 > Serial Number (system): T17H4CVN9D > Hardware UUID: 679E2FD9-24C7-5AC0-AECF-9753125F26EF > Provisioning UDID: 00006001-001C31521E69401E > Activation Lock Status: Disabled >

Kasper D. Hansen (21:58:37) (in thread): > Yeah, I can see why that is super weird.

Kasper D. Hansen (21:58:58) (in thread): > The compilers are not exactly the same > > clang-1500.1.0.2.4 > > vs > > clang-1500.0.40.1 >

Kasper D. Hansen (21:59:22) (in thread): > Other thing that comes to mind is I/O, since compilation can be very I/O dependent

Kasper D. Hansen (22:04:58) (in thread): > So on kjohnson3 you’re running “XCode 15.1 Beta 2” whereas on kjohnson1 you’re running Xcode 15.0 in either release version or one of the later beta versions, according tohttps://xcodereleases.com/alpha.html - Attachment (xcodereleases.com): Xcode Releases > More than you ever wanted to know™

Kasper D. Hansen (22:07:17) (in thread): > A quick googling doesn’t reveal anything about potential slowdowns with the 15.1 beta 2

Kasper D. Hansen (22:08:24) (in thread): > But perhaps this is a red flag. Most of the time spent in the build system is probably not on compiling

Hervé Pagès (22:14:17) (in thread): > Definitely not just a compilation issue. The CHECK times don’t involve any compilation at all (we use the--install=check:csaw.install-out.txttrick to avoid any re-installation) but it’s the same story e.g. forcsawit’s 1080.7s vs 513.7s.

Hervé Pagès (22:25:42) (in thread): > So we’re really puzzled by this. Jen suggested that there’s possibly something going on with the performance/efficiency cores e.g. when starting a job via a crontab (like we do for the builds) the OS would give it a lower priority and assign it preferrably to the efficiency cores which tend to be significantly slower. However there must be something more to the story because (1) this is no different from what we do on kjohnson1 and (2) nothing else is running on the machine, only the builds, which are highly parallelized and tend to put a lot of load on the machine, so it would be kind of surprising that the OS only sticks to the 8 efficiency cores in such situation.

Kasper D. Hansen (22:27:08) (in thread): > Have you tried just doing the build in a standard terminal for a package and then time it?

Kasper D. Hansen (22:27:32) (in thread): > At least that would be a point of comparison for the crontab hypothesis

Hervé Pagès (22:28:45) (in thread): > That’s the thing, if you go on kjohnson3 and try to build/checkcsawmanually, then it’s superfast! Which is why the crontab hypothesis is a good start but there’s more to that, it seems, I don’t know…

Kasper D. Hansen (22:29:05) (in thread): > hmm, ok, yeah that’s intriguing

Kasper D. Hansen (22:29:42) (in thread): > Have you tried asking Simon Urbanek? He sometimes has insane knowledge

Hervé Pagès (22:30:11) (in thread): > @Andres Wokaty? ^^^

2024-02-02

Andres Wokaty (11:00:13) (in thread): > It’s not ‘priority’ that’s being assigned, but ‘quality of service’, which is a range that determines whether something gets an efficiency core OR performance and efficiency cores. (You can’t–to my knowledge–assign just performance cores, at least via the terminal.) cron, which is how we start the builds, seems to be assigned to a lower quality of service then manually running a script for example because the latter will get a higher ‘clamp’, which is the lower number in the range. > > The build on kjohnson1 run from a cronjob will use performance cores. I thought this was because it only has 2 efficiency cores so maybe some threads get sent to the performance cores. > > I’ve been documenting my experiments with kjohnson3 athttps://github.com/Bioconductor/BBS/issues/387. If you run the build manually or via a script (not cron), which I am doing now performance improves to building every other day; however, it’s not currently robust. > > I will ask Simon about his experience. - Attachment: #387 kjohnson3 issues > The purpose of this issue is to collect all the information related to kjohnson3 not performing as we expect. > > System Information > > > # excerpt from system_profiler -detailLevel basic > > Software: > > System Software Overview: > > System Version: macOS 13.6.1 (22G313) > Kernel Version: Darwin 22.6.0 > Boot Volume: Macintosh HD > Boot Mode: Normal > Computer Name: kjohnson3 > User Name: (biocbuild) > Secure Virtual Memory: Enabled > System Integrity Protection: Enabled > Time since boot: 36 days, 23 hours, 27 minutes > > Storage: > > Data: > > Free: 1.76 TB (1,755,184,394,240 bytes) > Capacity: 2 TB (1,995,218,165,760 bytes) > Mount Point: /System/Volumes/Data > File System: APFS > Writable: Yes > Ignore Ownership: No > BSD Name: disk3s5 > Volume UUID: 84E902B4-385B-497B-8F88-8D55CA4A223E > Physical Drive: > Device Name: APPLE SSD AP2048Z > Media Name: AppleAPFSMedia > Medium Type: SSD > Protocol: Apple Fabric > Internal: Yes > Partition Map Type: Unknown > S.M.A.R.T. Status: Verified > > Macintosh HD: > > Free: 1.76 TB (1,755,184,394,240 bytes) > Capacity: 2 TB (1,995,218,165,760 bytes) > Mount Point: / > File System: APFS > Writable: No > Ignore Ownership: No > BSD Name: disk3s1s1 > Volume UUID: BEB26684-70B5-4407-9D99-FC9B003D9768 > Physical Drive: > Device Name: APPLE SSD AP2048Z > Media Name: AppleAPFSMedia > Medium Type: SSD > Protocol: Apple Fabric > Internal: Yes > Partition Map Type: Unknown > S.M.A.R.T. Status: Verified > > kjohnson3:~ biocbuild$ system_profiler SPHardwareDataType > Hardware: > > Hardware Overview: > > Model Name: Mac Pro > Model Identifier: Mac14,8 > Enclosure: Rack > Model Number: Z172000TRLL/A > Chip: Apple M2 Ultra > Total Number of Cores: 24 (16 performance and 8 efficiency) > Memory: 192 GB > System Firmware Version: 10151.41.12 > OS Loader Version: 8422.141.2.700.1 > Serial Number (system): G4NL9W9XNC > Hardware UUID: 360759C8-A272-58A9-97B9-B9F743AE7DB0 > Provisioning UDID: 00006022-001829260E90A01E > Activation Lock Status: Disabled > > > > kjohnson3:~ biocbuild$ sw_vers > ProductName: macOS > ProductVersion: 13.6.1 > BuildVersion: 22G313 > > > Tracking load > > We’re tracking the load at /Users/biocbuild/bbs-3.19-bioc-mac-arm64/log/uptime.log every 15 minutes. > > Performance installing csaw > > Indicators that it’s not performing as expected. @hpages compared the elapsed time of installing csaw. He thought possibly there might be an issue with the compiler, such as a regression. kjohnson1 (release) has clang-1500.0.40.1 and kjohnson3 has clang-1500.1.0.2.4. > > • https://bioconductor.org/checkResults/3.19/bioc-mac-arm64-LATEST/csaw/raw-results/kjohnson3/install-summary.dcf > > > Package: csaw > Version: 1.37.1 > Command: /Library/Frameworks/R.framework/Resources/bin/R CMD INSTALL csaw > StartedAt: 2023-12-28 16:11:13 -0500 (Thu, 28 Dec 2023) > EndedAt: 2023-12-28 16:46:06 -0500 (Thu, 28 Dec 2023) > EllapsedTime: 2092.9 seconds > RetCode: 0 > Status: OK > > > • https://bioconductor.org/checkResults/3.18/bioc-mac-arm64-LATEST/csaw/raw-results/kjohnson1/install-summary.dcf > > > Package: csaw > Version: 1.36.1 > Command: /Library/Frameworks/R.framework/Resources/bin/R CMD INSTALL csaw > StartedAt: 2023-12-30 10:42:36 -0500 (Sat, 30 Dec 2023) > EndedAt: 2023-12-30 10:44:10 -0500 (Sat, 30 Dec 2023) > EllapsedTime: 94.3 seconds > RetCode: 0 > Status: OK > > > Connection issues > > Connection Error on 11/13/23, 12/07/23, 12/20/23: > > > BBS> ============================================================== > BBS> (Re)make BBS_CENTRAL_BASEURL/products-in/kjohnson3/... OK > BBS> [STAGE2] STARTING STAGE2 at Wed Dec 20 17:24:55 2023 > Traceback (most recent call last): > File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/urllib/request.py", line 1346, in do_open > h.request(req.get_method(), req.selector, req.data, headers, > File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/http/client.py", line 1257, in request > self._send_request(method, url, body, headers, encode_chunked) > File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/http/client.py", line 1303, in _send_request > self.endheaders(body, encode_chunked=encode_chunked) > File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/http/client.py", line 1252, in endheaders > self._send_output(message_body, encode_chunked=encode_chunked) > File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/http/client.py", line 1012, in _send_output > self.send(msg) > File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/http/client.py", line 952, in send > self.connect() > File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/http/client.py", line 923, in connect > self.sock = self._create_connection( > File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/socket.py", line 843, in create_connection > raise err > File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/socket.py", line 831, in create_connection > sock.connect(sa) > OSError: [Errno 51] Network is unreachable > > During handling of the above exception, another exception occurred: > > Traceback (most recent call last): > File "/Users/biocbuild/BBS/BBS-run.py", line 785, in <module> > STAGE2() > File "/Users/biocbuild/BBS/BBS-run.py", line 397, in STAGE2 > BBSbase.waitForTargetRepoToBeReady() > File "/Users/biocbuild/BBS/BBSbase.py", line 75, in waitForTargetRepoToBeReady > f = urllib.request.urlopen(PACKAGES_url) > File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/urllib/request.py", line 214, in urlopen > return opener.open(url, data, timeout) > File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/urllib/request.py", line 517, in open > response = self._open(req, data) > File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/urllib/request.py", line 534, in _open > result = self._call_chain(self.handle_open, protocol, protocol + > File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/urllib/request.py", line 494, in _call_chain > result = func(*args) > File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/urllib/request.py", line 1375, in http_open > return self.do_open(http.client.HTTPConnection, req) > File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/urllib/request.py", line 1349, in do_open > raise URLError(err) > urllib.error.URLError: <urlopen error [Errno 51] Network is unreachable> > > > Other areas to explore > > • Any additional software added by IT

Kasper D. Hansen (11:11:05) (in thread): > I would be happy to reach out to Simon with a cc to you guys. I kind of know him from conferences, so sometimes that can help a bit

Kasper D. Hansen (11:11:35) (in thread): > Also, after sleeping on this, are there any power settings in macOS (potentially advanced power settings) which could be related?

Kasper D. Hansen (11:14:57) (in thread): > Is there a Battery mode in the systems settings - I know that sounds weird, but who knows?

Kasper D. Hansen (11:18:08) (in thread): > This smells like a similar issue:https://macperformanceguide.com/blog/2023/20230718_2200-MacPro2023M2Ultra-slow-mode.html

Kasper D. Hansen (11:21:59) (in thread): > You can apparently usepmsetto query the power management though the terminal, this is what I see on my laptop

Kasper D. Hansen (11:22:02) (in thread): > > $ pmset -g > System-wide power settings: > Currently in use: > standby 1 > Sleep On Power Button 1 > hibernatefile /var/vm/sleepimage > powernap 1 > networkoversleep 0 > disksleep 10 > sleep 1 (sleep prevented by powerd) > hibernatemode 3 > ttyskeepawake 1 > displaysleep 10 > tcpkeepalive 1 > lowpowermode 0 > womp 1 >

Kasper D. Hansen (11:22:39) (in thread): > (-gdisplays current settings). Perhaps wrap this in a crontab to see if somehow the server sleeps and it has trouble waking up?)

Kasper D. Hansen (11:22:58) (in thread): > You may have thought about all this, sorry if I am just blathering

Kasper D. Hansen (11:23:06) (in thread): > I’ll read the GH issue before I comment more

Andres Wokaty (11:26:04) (in thread): > I am fine reaching out to Simon. He helped me a lot when I was configuring the macs to build as cran does last year. Unless you mean that you might get a more favorable response from him than I would? (Sorry, I am not trying to step on toes or refuse help.) > > kjohnson3 is running ventura 13.6.1 . I will look at that post later, but I don’t think it’s a power issue. > > kjohnson3:~ biocbuild$ pmset -g > System-wide power settings: > DestroyFVKeyOnStandby 0 > Currently in use: > standby 0 > Sleep On Power Button 1 > autorestart 0 > powernap 1 > networkoversleep 0 > disksleep 10 > sleep 1 (sleep prevented by mds_stores, powerd, apsd) > ttyskeepawake 1 > displaysleep 10 > tcpkeepalive 1 > lowpowermode 0 > womp 1 >

2024-02-05

Clemens Kohl (08:13:17): > Hi everyone, > I am getting a install/build error for the packageAPLon kunpeng2 (https://bioconductor.org/checkResults/devel/bioc-LATEST/APL/kunpeng2-install.html) saying that the dependency Seurat is not available. The package builds fine on all other servers. Is there potentially a way to solve this problem on my end (besides the obvious one to remove all Seurat dependencies?) or can it be safely ignored? The build status plague on the landing page for Bioc devel says all is green/Okay. The full error is: > > ############################################################################## > ############################################################################## > ### > ### Running command: > ### > ### /home/biocbuild/R/R-devel_2024-01-16_r85812/bin/R CMD INSTALL APL > ### > ############################################################################## > ############################################################################## > > > * installing to library '/home/biocbuild/R/R-devel_2024-01-16_r85812/site-library' > ERROR: dependency 'Seurat' is not available for package 'APL' > * removing '/home/biocbuild/R/R-devel_2024-01-16_r85812/site-library/APL' > > Thanks for any help/advice!

Martin Grigorov (08:20:08): > Hi! This is a known problem with reticulate.There’sa discussion on the mailing list about from the last few days

Clemens Kohl (09:25:29) (in thread): > Thanks for pointing me to the tread! Despite me reading it I didnt connect it to my own problem. Am I assuming correctly that I can for now keep everything as it is and watch the github issue on reticulate for new development? Or should I implement some change to deal with this short term?

Martin Grigorov (09:29:25) (in thread): > The installation of Seurat fails due to leiden.Leiden fails due to reticulate

Clemens Kohl (09:36:06) (in thread): > got it, thanks!

2024-02-07

Luke Zappia (02:34:43): > I have this error for{zellkonverter}develonkunpeng2: ImportError: /usr/lib64/libstdc++.so.6: versionGLIBCXX_3.4.29’ not found (required by /home/biocbuild/.cache/R/basilisk/1.15.4/zellkonverter/1.13.2/zellkonverterAnnDataEnv-0.10.2/lib/python3.11/site-packages/pandas/_libs/window/aggregations.cpython-311-aarch64-linux-gnu.so). I fixed this on GitHub actions by runningsudo apt-get upgrade libstdc++6`, would it be possible to do something similar here?

Martin Grigorov (02:35:32) (in thread): > Let me take a look!

Martin Grigorov (02:39:14) (in thread): > it provides GLIBCXX_3.4.28 …

Martin Grigorov (02:42:23) (in thread): > I was able to installgcc-toolset-12-libstdc++-12.2.1-14.aarch64that provides 3.4.30

Martin Grigorov (02:45:44) (in thread): > > biocbuild@kunpeng2 ~/git> set -x LD_LIBRARY_PATH $LD_LIBRARY_PATH /opt/openEuler/gcc-toolset-12/root/usr/lib64/ > biocbuild@kunpeng2 ~/git> R CMD build zellkonverter > * checking for file 'zellkonverter/DESCRIPTION' ... OK > * preparing 'zellkonverter': > * checking DESCRIPTION meta-information ... OK > * installing the package to build vignettes > * creating vignettes ... OK > * checking for LF line-endings in source and make files and shell scripts > * checking for empty or unneeded directories > Omitted 'LazyData' from DESCRIPTION > * building 'zellkonverter_1.13.2.tar.gz' > > Success !

Martin Grigorov (02:46:41) (in thread): > it should be green in the next report ! Or the one after the next

Luke Zappia (02:47:02) (in thread): > :tada:Thanks for responding so quickly!

Martin Grigorov (02:47:43) (in thread): > Welcome!:slightly_smiling_face:

Martin Grigorov (03:53:17) (in thread): > btw one of the tests fails: > > Running 'testthat.R' > ERROR > Running the tests in 'tests/testthat.R' failed. > Last 13 lines of output: > ── Error ('test-write.R:102:5'): writeH5AD works as expected with version 0.7.6 ── > Error: one or more Python packages failed to install [error code 1] > Backtrace: > ▆ > 1. └─zellkonverter::writeH5AD(sce, temp, version = "0.7.6") at test-write.R:102:5 > 2. └─basilisk::basiliskRun(...) > 3. └─basilisk::basiliskStart(...) > 4. └─basilisk::obtainEnvironmentPath(env) > 5. └─basilisk::setupBasiliskEnv(...) > 6. └─reticulate::conda_install(...) > 7. └─reticulate:::stopf(fmt, result) > > [ FAIL 2 | WARN 28 | SKIP 3 | PASS 125 ] > Error: Test failures > Execution halted > * checking for unstated dependencies in vignettes ... OK > * checking package vignettes ... OK > * checking re-building of vignette outputs ... OK > * checking PDF version of manual ... OK > * DONE > > Status: 1 ERROR, 1 NOTE > See > '/home/biocbuild/git/zellkonverter.Rcheck/00check.log' > for details. >

Martin Grigorov (03:54:29) (in thread): > it is not clear which Python package(s) is not installed

Luke Zappia (04:06:12) (in thread): > Hmm…I guess that environment didn’t build correctly for some reason? Maybe the cache for that needs to be cleared so it’s rebuilt?

Martin Grigorov (04:07:13) (in thread): > which cache ?

Luke Zappia (04:09:00) (in thread): > ****{basilisk}****creates the Python environment in a cache directory. Of the top of the end I don’t remember the exact path but it’s something like/cache_dir/basilisk_version/zellkonverter_version/envrionment_version.

Martin Grigorov (04:11:25) (in thread): > I see there is something at~/.cache/R/basilisk/1.15.4/zellkonverter/1.13.2/zellkonverterAnnDataEnv-0.10.2/I will remove~/.cache/R/basilisk

Luke Zappia (04:12:26) (in thread): > That will probably reset the cache for all packages the use****{basilisk},****maybe just removing thezellkonvertersub-folder would be enough for this package

Martin Grigorov (04:12:39) (in thread): > OK !

Martin Grigorov (05:06:39) (in thread): > It didn’t help > I removed/.cache/R/basilisk/*/zellkonverter/but it still fails the same way

Martin Grigorov (05:06:42) (in thread): > > ll ~/.cache/R/basilisk/1.15.4/zellkonverter/1.13.2/ > total 12K > drwxr-xr-x 15 biocbuild biocbuild 4.0K Feb 7 09:20 zellkonverterAnnDataEnv-0.10.2 > drwxr-xr-x 14 biocbuild biocbuild 4.0K Feb 7 09:34 zellkonverterAnnDataEnv-0.8.0 > drwxr-xr-x 15 biocbuild biocbuild 4.0K Feb 7 09:29 zellkonverterAnnDataEnv-0.9.2 >

Vince Carey (06:43:12) (in thread): > When I run into problems with python interop I tend to remove all of .cache/R/basilisk. It does lead to additional reconstruction efforts in next build but it provides a cleaner basis for diagnosis. Also of some note is that setting BASILISK_MINICONDA_VERSION to py311_23.11.0-2 introduces the mamba dependency resolution algorithm (I think) which gives faster package version identification and clearer messages on incompatibilities. We can’t adopt this uniformly because non-interactive windows installation is broken (reported and as yet unaddressed) but it can be helpful for diagnostics.

Vince Carey (06:44:21) (in thread): > (We can’t adopt the py311 miniconda uniformly, and thus we don’t adopt it at all :(.)

Martin Grigorov (06:59:38) (in thread): > I backed up~/.cache/R/basilisk,export BASILISK_MINICONDA_VERSION=py311_23.11.0-2and re-ran the check for zellkonverter > Let’s see !

Martin Grigorov (07:23:32) (in thread): > still the same:confused:

2024-02-11

Clemens Kohl (11:02:43): > Hi everyone, sorry for the many messages, but I am was fixing a bug in the current Bioconductor Release 3.18 for the package APL. The bug is fixed but now it seems like there is (and probably was before) a dependency missing on the MacOS serverlconway: APL build reportReticulate cannot loadtorch: > > ModuleNotFoundError: No module named 'torch' > > Would it be possible to installpytorchon the machine?

2024-02-12

Alan Murphy (03:41:21): > Hey! I noticed in the builds thatkunpeng2 is missing at least one package dependency needed to test my package (SNPlocs.Hsapiens.dbSNP155.GRCh37is missing), could this be installed on it?

Lori Shepherd (07:33:58) (in thread): > What is the name of your package? And that dependency is listed as a suggest in the DESCRIPTION of your package?

Alan Murphy (07:34:51) (in thread): > MungeSumstats, it’s a suggestion but that was the advised approach given the size of these reference packages

Lori Shepherd (07:37:41) (in thread): > @Martin Grigorovany ideas why the package wouldn’t be installed automatically as a valid Bioconductor annotation package?

Martin Grigorov (07:38:10) (in thread): > Let me see!

Martin Grigorov (07:39:58) (in thread): > > BiocManager::install("SNPlocs.Hsapiens.dbSNP155.GRCh37") > Bioconductor version 3.19 (BiocManager 1.30.22), R Under development (unstable) > (2024-01-16 r85812) > Installing package(s) 'SNPlocs.Hsapiens.dbSNP155.GRCh37' > trying URL '[https://bioconductor.org/packages/3.19/data/annotation/src/contrib/SNPlocs.Hsapiens.dbSNP155.GRCh37_0.99.24.tar.gz](https://bioconductor.org/packages/3.19/data/annotation/src/contrib/SNPlocs.Hsapiens.dbSNP155.GRCh37_0.99.24.tar.gz)' > Content type 'application/x-gzip' length 6048515545 bytes (5768.3 MB) > > I guess it timed out

Martin Grigorov (07:40:57) (in thread): > recently I updated the R-devel version, as requested by@Andres Wokaty. I think I installed all such big packages which timeout during normal build but it seems this one still failed … I will install it now!

Lori Shepherd (07:41:48) (in thread): > thanks!

Martin Grigorov (07:42:20) (in thread): > Thanks for the ping! I’ve missed the message by Alan

2024-02-13

Andres Wokaty (11:03:46) (in thread): > I’ve installpytorchon lconway. You should see that error resolve on tomorrow’s report.

Clemens Kohl (11:28:45) (in thread): > Thanks a lot!

Andres Wokaty (15:03:44): > Maintenance will be performed on Nebbiolo1 and Nebbiolo2 on Wednesday 2/14, so build reports may not start again until Friday 2/16. Thanks for your patience.

Jianhai Zhang (20:29:38): > @Jianhai Zhang has joined the channel

Abdullah Al Nahid (21:34:37): > @Abdullah Al Nahid has joined the channel

2024-02-14

Martin Grigorov (02:31:47): > @Jianhai ZhangPlease read this email thread about the issue with Seurat (leiden and reticulate) -https://stat.ethz.ch/pipermail/bioc-devel/2024-February/020161.html

Martin Grigorov (02:32:19) (in thread): > I will try to reproduce it in a Docker container and report it to the maintainer ofreticulate

Alan Murphy (05:05:23) (in thread): > Hey!SNPlocs.Hsapiens.dbSNP155.GRCh38also seems to be missing (again I would say it’s a size issue). Probably worth checking if the following are installed too: > > BSgenome.Hsapiens.1000genomes.hs37d5 > BSgenome.Hsapiens.NCBI.GRCh38 > SNPlocs.Hsapiens.dbSNP144.GRCh37 > SNPlocs.Hsapiens.dbSNP144.GRCh38 >

Martin Grigorov (05:10:06) (in thread): > > package(s) not installed when version(s) same as or greater than current; use > `force = TRUE` to re-install: 'BSgenome.Hsapiens.1000genomes.hs37d5' > > package(s) not installed when version(s) same as or greater than current; use > `force = TRUE` to re-install: 'BSgenome.Hsapiens.NCBI.GRCh38' > > package(s) not installed when version(s) same as or greater than current; use > `force = TRUE` to re-install: 'SNPlocs.Hsapiens.dbSNP144.GRCh37' > > SNPlocs.Hsapiens.dbSNP144.GRCh38- installed earlier today

Martin Grigorov (05:10:18) (in thread): > all is good!

Martin Grigorov (05:12:30) (in thread): > I was not able to reproduce it in a Docker container …

Vince Carey (06:00:56) (in thread): > Hi Martin – what is the output ofreticulate::py_config()?

Martin Grigorov (06:02:41) (in thread): > > reticulate::py_config() > > ***** caught segfault ***** > address 0x90, cause 'memory not mapped' > > Traceback: > 1: py_initialize(config$python, config$libpython, config$pythonhome, config$virtualenv_activate, config$version >= "3.0", interactive(), numpy_load_error) > 2: (function() { Sys.setenv(PYTHONPATH = newpythonpath) on.exit(Sys.setenv(PYTHONPATH = oldpythonpath), add = TRUE) py_initialize(config$python, config$libpython, config$pythonhome, config$virtualenv_activate, config$version >= "3.0", interactive(), numpy_load_error)})() > 3: doTryCatch(return(expr), name, parentenv, handler) > 4: tryCatchOne(expr, names, parentenv, handlers[[1L]]) > 5: tryCatchList(expr, classes, parentenv, handlers) > 6: tryCatch({ oldpythonpath <- Sys.getenv("PYTHONPATH") newpythonpath <- Sys.getenv("RETICULATE_PYTHONPATH", unset = paste(config$pythonpath, system.file("python", package = "reticulate"), sep = .Platform$path.sep)) local({ Sys.setenv(PYTHONPATH = newpythonpath) on.exit(Sys.setenv(PYTHONPATH = oldpythonpath), add = TRUE) py_initialize(config$python, config$libpython, config$pythonhome, config$virtualenv_activate, config$version >= "3.0", interactive(), numpy_load_error) })}, error = function(e) { Sys.setenv(PATH = oldpath) if (is.na(curr_session_env)) { Sys.unsetenv("R_SESSION_INITIALIZED") } else { Sys.setenv(R_SESSION_INITIALIZED = curr_session_env) } stop(e)}) > 7: initialize_python() > 8: ensure_python_initialized() > 9: reticulate::py_config() > > Possible actions: > 1: abort (with core dump, if enabled) > 2: normal R exit > 3: exit R without saving workspace > 4: exit R saving workspace >

Martin Grigorov (06:03:06) (in thread): > there is something wrong with Reticulate onkunpeng2

Vince Carey (06:03:34) (in thread): > You might be able to take steps with environment variable settings to get your build environment to replicate that of the container. RETICULATE_PYTHON can be set to a specific python binary.

Martin Grigorov (06:04:19) (in thread): > > biocbuild@kunpeng2 ~/git [71]> echo $RETICULATE_PYTHON > /usr/bin/python3 > biocbuild@kunpeng2 ~/git> $RETICULATE_PYTHON --version > Python 3.9.9 >

Vince Carey (06:05:28) (in thread): > what’s the setting (if any) in the container? i am guessing that you have an ARM linux container with R for which reticulate is working, we would want to ensure that the kungpeng2 builder’s R uses a compatible python

Martin Grigorov (06:07:07) (in thread): > the container is still installing Seurat (BiocManager::install("Seurat")). I will check once it finish

Martin Grigorov (08:54:28) (in thread): > > reticulate::py_config() > python: /root/.virtualenvs/r-reticulate/bin/python > libpython: /usr/lib/python3.10/config-3.10-aarch64-linux-gnu/libpython3.10.so > pythonhome: /root/.virtualenvs/r-reticulate:/root/.virtualenvs/r-reticulate > version: 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0] > numpy: /root/.virtualenvs/r-reticulate/lib/python3.10/site-packages/numpy > numpy_version: 1.26.4 > numpy: /root/.virtualenvs/r-reticulate/lib/python3.10/site-packages/numpy > > @Vince Careythis is from the container

Martin Grigorov (08:54:46) (in thread): > I will try removing the RETICULATE_PYTHON env var on kunpeng2

Martin Grigorov (08:56:02) (in thread): > without the env var it works: > > reticulate::py_config() > python: /home/biocbuild/.virtualenvs/r-reticulate/bin/python > libpython: /usr/lib64/libpython3.9.so > pythonhome: /home/biocbuild/.virtualenvs/r-reticulate:/home/biocbuild/.virtualenvs/r-reticulate > version: 3.9.9 (main, Oct 18 2023, 18:17:13) [GCC 10.3.1] > numpy: /home/biocbuild/.virtualenvs/r-reticulate/lib/python3.9/site-packages/numpy > numpy_version: 1.26.3 >

Martin Grigorov (08:57:47) (in thread): > leidenhas been installed! NowSeurat!

Martin Grigorov (09:00:52) (in thread): > Seurat is also installed ! Hooray!

Vince Carey (18:56:53) (in thread): > Great. Does this imply some alteration to the BBS recommendations on management of python?

2024-02-15

Martin Grigorov (08:13:12) (in thread): > I followed the recomendation athttps://github.com/Bioconductor/BBS/blob/devel/Doc/Prepare-Ubuntu-22.04-HOWTO.md#set-reticulate_python-in-etcprofileBut for some reason the system Python does not work anymore (I think the problem started with R 4.4/devel and Bioc 3.19 > I don’t know what exactly is the problem though. Any reticulate operation fails with segfaults

2024-02-16

Jovana Maksimovic (03:01:20): > @Jovana Maksimovic has joined the channel

2024-02-21

Henrik Bengtsson (14:13:32): > Saw the “[Bioc-devel] Fw: MungeSumstats Bioconductor” thread where@Hervé Pagèscommented “These random kills are a desperate attempt by the Linux kernel to keep the machine alive when it’s under extremely high load and running out of resources”. > > Has there been a discussion on configuring CGroups so that each build/check process is allotted a certain amount of RAM and number of CPU cores? CGroups would prevent “misbehaving” processes from breaking out of this sandbox and negatively affect other processes on the same machine. If a process would consume too much RAM, the Linux out-of-memory (OOM) killer would “take care” of that process.

Andres Wokaty (14:55:31) (in thread): > I don’t think so. Herve’s out for the week. We can discuss when he returns.

2024-02-22

Mike Smith (15:22:17): > Are the Bioc 3.19 Mac ARM64 builds on hold for now? The build report (https://bioconductor.org/checkResults/3.19/bioc-mac-arm64-LATEST/) doesn’t seem to have take a new snapshot or updated in about a month.

Andres Wokaty (15:33:45) (in thread): > Yes, sorry, I’m still having issues with the machine and have to coordinate the effort with IT to troubleshoot it.

2024-02-23

Mike Smith (04:54:36) (in thread): > Thanks, I was asking because I made a few changes to my packages to try and getting them to finish within the time limit, but they haven’t been built since then. I wasn’t sure if the slow speeds meant it was just running less frequently, or it was out of commission for the time being. This low priority cron task thing sounds like a major pain!

Andres Wokaty (14:24:02): > Hi, the devel builders will be running on a Monday, Wednesday, Friday schedule and producing reports for Linux, Mac Intel, and Windows on those days. This will accommodate more packages and allow slower builders to complete a full cycle.

2024-02-27

Vince Carey (06:41:26): > Any thoughts in the community on whether cross-compilation would help us accelerate testing and packaging for, e.g., apple silicon?https://clang.llvm.org/docs/CrossCompilation.html– any experimenters? data?

Vince Carey (07:04:41): > I ask because we have access to abundant x86/linux platforms but big disappointments with newly acquired apple hardware

Aaron Lun (11:09:06): > Based on cross-compiling experience for WebAssembly and in attempting Arm64 builds on the GH actions: for any sophisticated C/C++ project (e.g., HDF5), it is a pathway to much suffering. This is because these large projects execute their binaries duringconfigureand this fails in a cross-compiling context. Some trickery is required to fool the compilation to completion, e.g., the Wasm binaries are executed via Node.js by overriding how CMake calls the binaries. Never figured out how to do that for Mac Arm so I just ended up providing pre-built binaries for linking (e.g.,https://github.com/ArtifactDB/dolomite-base/blob/master/.github/workflows/publish-pypi.yml).

Alan O’C (12:57:03): > My experience with cross-compilation for image analysis libraries is also one of immense pain

Alan O’C (12:57:49): > Not an ideal long term solution but the github macos14 runners are mac arm, if extra capacity would be useful

2024-02-28

Mike Smith (09:04:34): > I’ll just echo what the other have said, my experience with cross-compiling has been frustrating and/or fruitless once the package in question gets moderatly complex. As an example of what Aaron mentioned, here’s where the R-universe build system failed when trying to cross compile Rhdf5lib for Mac ARM64 (https://github.com/r-universe/grimbough/actions/runs/7813284750/job/21313329596#step:7:389). That specific example probably goes away now GitHub offers ARM64 runners, but the principle is the same

Lluís Revilla (09:09:57): > Yes,@Jeroen Oomsmight have more experience on cross-compiling packages at r-universe, Posit people might have also experiences (Gabor Csardi probably) or CRAN volunteers…

Jeroen Ooms (09:11:19): > @Jeroen Ooms has joined the channel

Jeroen Ooms (09:16:47): > I found that cross compiling generally works well, indeed, except for R packages that bundle embedded libraries and try to autoconf those on the fly. On CRAN the general advice is to avoid bundling libraries and instead rely on the distro (apt,yum) or toolchain bundle (rtools, or the cran macos-libs) for providing the external libraries. See also:https://ropensci.org/blog/2024/01/14/runiverse-arm64/ - Attachment (ropensci.org): R-universe now builds MacOS ARM64 binaries for use on Apple Silicon (aka M1/M2/M3) systems > R-universe now builds MacOS ARM64 binaries for use on Apple Silicon (aka M1/M2/M3) systems.

Andres Wokaty (09:37:29) (in thread): > Thanks for sharing your process!

2024-02-29

Andres Wokaty (11:59:21) (in thread): > kjohnson3 was able to run a full build yesterday:https://bioconductor.org/checkResults/3.19/bioc-mac-arm64-LATEST/. It’s really too early to say if its issues are resolved, but I am trying to set it to follow the other devel builders.

2024-03-01

Hervé Pagès (05:21:28) (in thread): > Never heard of CGroups before. We can look into it. Thanks@Henrik Bengtssonfor the suggestion. > BTW no more kills by the OOM killer on our Linux builders since we increased the swap space from 8G to 24G:smiley:

2024-03-02

Aaron Lun (06:27:19): > FYI the most recenthttps://master.bioconductor.org/checkResults/3.19/bioc-LATEST/scran/nebbiolo1-checksrc.htmlis complaining about missingmonoclebut AFAICTmonocleis still bulding on the BBS. Not sure what’s going on here. > > * using log directory ‘/home/biocbuild/bbs-3.19-bioc/meat/scran.Rcheck’ > * using R Under development (unstable) (2024-01-16 r85808) > * using platform: x86_64-pc-linux-gnu > * R was compiled by > gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0 > GNU Fortran (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0 > * running under: Ubuntu 22.04.3 LTS > * using session charset: UTF-8 > * checking for file ‘scran/DESCRIPTION’ ... OK > * this is package ‘scran’ version ‘1.31.2’ > * checking package namespace information ... OK > * checking package dependencies ... ERROR > Package suggested but not available: ‘monocle’ > > The suggested packages are required for a complete check. > Checking can be attempted without them by setting the environment > variable *R_CHECK_FORCE_SUGGESTS* to a false value. > > See section ‘The DESCRIPTION file’ in the ‘Writing R Extensions’ > manual. >

Lori Shepherd (07:20:30): > Actuallymonocle isn’tinstalling/buildingin develhttps://bioconductor.org/checkResults/devel/bioc-LATEST/monocle/

Lori Shepherd (07:21:06): > We reached out to the maintainer but they have not responded yet

Lori Shepherd (07:35:21): > The qlcMatrix package was removed from CRAN and monocle needs to be adjusted. FWIW I expect this will start to appear in release too as soon as we reinstall the latest R 4-3 patch on release bioc as well.https://cran.r-project.org/web/packages/qlcMatrix/index.html

Aaron Lun (17:25:35): > oh well. i just removed the monocle dependency

Aaron Lun (17:25:37): > ¯*(ツ)*/¯

2024-03-04

Alan O’C (08:26:02): > Looking at the github issues it seems to be abandonware at this point

Alan O’C (08:28:01): > Might be good to encourage archiving/deprecating old packages when introducing a new version that supercedes it

Hervé Pagès (13:05:08): > They didn’t submitmonocle3to Bioconductor yet. However, regardless of whether they intend to submit or not,monoclewillbe deprecated in 3.19 if it doesn’t get fixed in time for the release. This is standard procedure.

Alan O’C (18:31:22): > Ah okay, that makes sense. I assumed it was a DESeq/DESeq2 kinda job. Yeah makes sense, does not inspire confidence, though they’re hardly the only sinners in that regard

2024-03-05

Vamika Mendiratta (02:08:42): > @Vamika Mendiratta has joined the channel

Benedicta Imelda (17:48:43): > @Benedicta Imelda has joined the channel

Hervé Pagès (18:58:59) (in thread): > Big difference withDESeq/DESeq2is thatDESeqwas always green on all platforms so we (the core team) had no reasons to kick it out. In the case ofmonocle, it’s broken on all platforms (people can’t even install it anymore).

2024-03-07

Constantin Ahlmann-Eltze (03:27:13): > Hi, my package glmGamPoi has been failing on Iconway (mac) for a few weeks:https://bioconductor.org/checkResults/release/bioc-LATEST/glmGamPoi/lconway-buildsrc.htmlThe error is specific to Mac and also does not occur in the devel version. The error message states that the functions ‘colData’ and ‘counts’ are not found for signature ‘tbl_df’ (even though I explicitly load the SingleCellExperiment packagehttps://code.bioconductor.org/browse/glmGamPoi/blob/RELEASE_3_18/vignettes/pseudobulk.Rmd#L39). > > I failed to reproduce the problem on my MacBook (Big Sur) with R 4.3.2 Is it possible that calling ‘muscData::Kang18_8vs8()’ on the build system returns a tibble instead of a SingleCellExperiment? And how would I debug this? - Attachment (code.bioconductor.org): Bioconductor Code: glmGamPoi > Browse the content of Bioconductor software packages.

Mike Smith (04:43:46) (in thread): > I don’t have an answer to your query, but I am confused why the Iconway report doesn’t appear at all on the build summary for the (https://bioconductor.org/checkResults/release/bioc-LATEST/glmGamPoi/). It’s only there if you explicitly follow the link you provided.

Constantin Ahlmann-Eltze (07:42:03) (in thread): > Huh, interesting. It looks like kjohnson1 replaced Iconway?! Anyways, on my intel mac there still seem to be no binaries available

Vince Carey (07:48:54) (in thread): > I had no problem with build and check in devel on my M1 mac. We will be looking at this today.

Constantin Ahlmann-Eltze (07:50:15) (in thread): > Thanks. I just saw that the latest build on devel complains about about a corrupted cache:https://bioconductor.org/checkResults/devel/bioc-LATEST/glmGamPoi/lconway-buildsrc.html

Lori Shepherd (08:33:42) (in thread): > we’ll look into resolving the cache error that is occurring on lconway

Lori Shepherd (09:07:45) (in thread): > FWIW: kjohnson1 didnt replace lconway – kjohnson is the mac arm64 builder while lconway/merida1 are the intel mac builds – we recently swapped the lconway and merida1 mac machines so it could be some of the issue/confusion.

Constantin Ahlmann-Eltze (09:08:57) (in thread): > Ah, but is it on purpose then thathttps://bioconductor.org/checkResults/release/bioc-LATEST/glmGamPoi/does not list Iconway/merida1 anymore?

Lori Shepherd (09:09:39) (in thread): > we recently swapped the machines and are making sure they are configured and building correctly before adding them back into the reports. They should be back within the next few day.

Constantin Ahlmann-Eltze (09:10:01) (in thread): > I see. That makes sense:slightly_smiling_face:

Lori Shepherd (09:10:46) (in thread): > as far as no binaries available – that is also because we were updating the builders for 1. the new R 4.3 patched version released last week and 2. updating R devel (which we do on occasion when the builders are using R-devel to stay up-to-date)

Lori Shepherd (09:11:12) (in thread): > so once the macs run and propagate it should all be straightened out automatically

Constantin Ahlmann-Eltze (09:11:56) (in thread): > Perfect. Then I will just check again sometime next week and keep my fingers crossed:slightly_smiling_face:Thanks for the explanations:slightly_smiling_face:

Lori Shepherd (09:12:22) (in thread): > I also fixed the cache error on lconway so that should be cleared now

2024-03-09

Cherishma Subhasa (20:18:47): > @Cherishma Subhasa has joined the channel

2024-03-11

Constantin Ahlmann-Eltze (04:47:08) (in thread): > Hi, I just wanted to quickly confirm that the build errors are all fixed now. Thanks again:slightly_smiling_face:

2024-03-17

Raihanat Adewuyi (20:42:56): > @Raihanat Adewuyi has joined the channel

2024-03-18

Hervé Pagès (19:28:38): > @Aaron Lun@Lori Shepherd@Alan O’CLooks likemonoclegot fixed:https://bioconductor.org/checkResults/3.19/bioc-LATEST/monocle/:blob_cheer:

2024-03-20

Eszter Ari (04:40:55): > @Eszter Ari has joined the channel

Eszter Ari (04:47:36): > Hi, I have a question about “a problem reported in the Build/check report for BioC 3.19 experimental data”: > I have created an experimental data package:https://github.com/ELTEbioinformatics/muleaDataI have got this message fromBBS-noreply@bioconductor.org: > According to the Build/check report for BioC 3.19 experimental data, > the muleaData package has the following problem(s): > > o ERROR for ‘R CMD check’ on nebbiolo1. See the details here:https://master.bioconductor.org/checkResults/3.19/data-experiment-LATEST/muleaData/nebbiolo1-checksrc.htmlAs I see on the link there is a problem with the muleaData.Rmd file. Previously I have been asked by@Lori Shepherdto make the ‘install’ R code chunk to eval=FALSE:https://github.com/Bioconductor/Contributions/issues/3291#issuecomment-1978971419 > >{‘install’, eval=FALSE} > if (!require(“BiocManager”, quietly = TRUE)) > install.packages(“BiocManager”) > > BiocManager::install(“ExperimentHub”) > BiocManager::install(“muleaData”) > > > Then the next ‘example’ chunk cannot be run: > >{‘example’} > > # Calling the ExperimentHub library. > library(ExperimentHub) > > # Downloading the metadata from ExperimentHub. > eh <- ExperimentHub() > > # Creating the muleaData variable. > muleaData <- query(eh, “muleaData”) > > # Checking the muleaData variable. > muleaData > > # Looking for the ExperimentalHub ID of i.e. target genes of transcription > # factors from TFLink in Caenorhabditis elegans. > mcols(muleaData) %>% > as.data.frame() %>% > dplyr::filter(species == “Caenorhabditis elegans” & > sourceurl == “https://tflink.net/”) > > # Creating a variable for the GMT data.frame of EH8735. > # EH8735 contains small-scale measurement results, where the target genes are > # coded with Ensembl ID-s > Transcription_factor_TFLink_Caenorhabditis_elegans_SS_EnsemblID <- > muleaData[[“EH8735”]] > > > Should I change to eval=FALSE this chunk as well? - Attachment: Comment on #3291 muleaData ExperimentHubData Bioconductor package

Eszter Ari (05:10:18) (in thread): > I also found this:https://master.bioconductor.org/packages/devel/bioc/vignettes/BiocCheck/inst/doc/BiocCheck.html#vignette-checks“Checks whether the number of eval=FALSE chunks is more than 50% of the total (WARNING). > Checks whether the global vignette code option is set to eval=FALSE. The majority of vignette code is expected to be evaluated (WARNING)” > > So can I set all code chunks in the vignette to eval=FALSE then or not?

Eszter Ari (05:15:04) (in thread): > I was suggested to add ExperimentHub and dplyr as suggested packages to the DEASCRIPTION file by@Mike Smith. I try it!

Alan O’C (06:26:41) (in thread): > Why can’t the “example” chunk be run?

Lori Shepherd (07:23:23) (in thread): > as@Mike Smithsuggested if you did not have ExperimentHub and dplyr in your DESCRIPTION as Suggests (or Depends or Imports) it would be needed there. It is related tohttps://stat.ethz.ch/pipermail/bioc-devel/2023-March/019555.htmlwhere all needed dependencies are checked. Vignette code should be run and you should NOT change to eval=FALSE.

Eszter Ari (07:25:57) (in thread): > Thank you@Lori Shepherd

2024-03-24

Najla Abassi (09:56:11): > @Najla Abassi has joined the channel

2024-03-26

Maximilian Mattessich (14:54:06): > @Martin Grigorovapologies if you’re not the person to contact about this - has there been any further development on the issues with Seurat on kunpeng2 discussed onthis mailing list thread? > We are currently running into the same missing package build error with Seurat only on kunpeng2. We tried adding Seurat to ‘Suggests’ in our DESCRIPTION file, but this doesn’t fix the error. Are there any other fixes we could try to avoid this error on release? > > For reference here is the build error (alsolinked here): > > * checking for file 'nipalsMCIA/DESCRIPTION' ... OK > * preparing 'nipalsMCIA': > * checking DESCRIPTION meta-information ... OK > * installing the package to build vignettes > * creating vignettes ... ERROR > --- re-building 'Vignette1.Analysis-of-MCIA-Decomposition.Rmd' using rmarkdown > --- finished re-building 'Vignette1.Analysis-of-MCIA-Decomposition.Rmd' > > --- re-building 'Vignette2.Single-Cell-Analysis.Rmd' using rmarkdown > > Quitting from lines 59-72 [load-packages] (Vignette2.Single-Cell-Analysis.Rmd) > Error: processing vignette 'Vignette2.Single-Cell-Analysis.Rmd' failed with diagnostics: > there is no package called 'Seurat' > --- failed re-building 'Vignette2.Single-Cell-Analysis.Rmd' >

Martin Grigorov (14:57:18) (in thread): > Hi! I’m the right person!:blush:The issue with Seurat have been fixed few months ago but maybe something else broke with the new update of R-devel last week > I will investigate tomorrow!

Maximilian Mattessich (15:02:15) (in thread): > Great - thanks for the quick reply!

Aaron Lun (21:00:33): > trying to debug some odd errors withhttps://bioconductor.org/checkResults/devel/bioc-LATEST/iSEE/nebbiolo1-checksrc.html; do the linux builders have any additional process that deletes files in~/.cache? It’s the only explanation I can think of for the build error, given that the function call fails in the examples but the same call succeeds in the tests.

2024-03-27

Martin Grigorov (05:31:24) (in thread): > @Maximilian MattessichThe issue is fixed (again)!

Andres Wokaty (09:37:41) (in thread): > No, but I do sometimes delete the cache when installing a new R, which I did recently on the devel builders.

Aaron Lun (10:51:24) (in thread): > oh, okay. hopefully the failure will resolve itself in the next build.

2024-03-28

Alan Murphy (07:59:49) (in thread): > Hey! There seems to be the same issue with installed packages in release for kunpeng2 (linux):https://bioconductor.org/checkResults/3.19/bioc-LATEST/MungeSumstats/kunpeng2-buildsrc.html

Martin Grigorov (08:51:04) (in thread): > Let me check !

Martin Grigorov (08:53:02) (in thread): > 6GB are being downloaded!

Martin Grigorov (08:53:41) (in thread): > I have a script that gives me a list of missing packages that break INSTALL. I will need to update it to add the packages forbuildtoo!

Martin Grigorov (09:01:48) (in thread): > the problem is the non-standard error in your package: > > Error: processing vignette 'MungeSumstats.Rmd' failed with diagnostics: > Install 'SNPlocs.Hsapiens.dbSNP155.GRCh37' to use 'GRCh37' as 'ref_genome' and 155 as dbSNP version > > Usually the errors from the other packages look like: > > Skip segmentSeq, due to not available for package ERROR: dependency 'baySeq' is not available for package 'segmentSeq' > Skip SeqSQC, due to not available for package ERROR: dependency 'rbokeh' is not available for package 'SeqSQC' > > and my script has logic to parse the names of the dependencies > I guess your package uses a custom way to fetch the dependency and thus the custom error message

Alan Murphy (09:12:22) (in thread): > Yeah it’s because we put those packages as suggests but we need them to run certain functions so I thought the custom messaging was useful

Maximilian Mattessich (12:08:54) (in thread): > It looks like our package builds now.Thanks for your help!

2024-04-02

Alan Murphy (05:27:38) (in thread): > Hey@Martin Grigorovjust noticed it’s failing on the same machine but now because SNPlocs.Hsapiens.dbSNP155.GRCh38 isn’t installed (as of 1st April), not sure if you have that added now too and there is just a delay?

Martin Grigorov (05:34:50) (in thread): > I’llinstall it tomorrow!

Alan Murphy (05:35:01) (in thread): > Thank you!

2024-04-03

Martin Grigorov (05:20:44) (in thread): > SNPlocs.Hsapiens.dbSNP155.GRCh38has been installed !

Martin Grigorov (05:21:21) (in thread): > please let me know if there are more missing packages !

Alan Murphy (05:21:35) (in thread): > Thank you Martin!! Will do

Matteo Tiberti (05:51:23): > hi everyone, I read in the 3.19 release deadline email that “The deadline for devel Bioconductor 3.19 for packages to pass R CMD build and R CMD check is Friday April 26th”. Currently, we have a couple of fails on macOS arm64 only (they are segmentation faults - including on a package, MoonlightR, that we haven’t changed since 3.18 I believe). I have a couple of questions: > * is the deadline also valid for Mac arm64 builds? I’m asking because reports are on a separate page respect to the other builds, so I was wondering. This might just due because it’s run less frequently > * what happens if we still have a fail in MacOS arm64 by the deadline? > We’ll do our best to have the build fixed, but I just don’t have an Arm64 Mac available right now to debug, and the builders are run less frequently than for x86, so I’m a bit worried we might not make it on time and would rather plan in advance > > thanks!

Martin Grigorov (05:53:42) (in thread): > Do you have experience with Github Actions CI ? > They provide free macOS arm64 runners where you can debug via logging

Lori Shepherd (06:46:22) (in thread): > Pleasedo your best to fix the packageon arm64 but the package will not be excluded if that is there only platform it is failing on.

Matteo Tiberti (07:33:11) (in thread): > Thanks@Martin Grigorov- we don’t really have experience with that but would be good to check out (for this and other reasons) > ok@Lori Shepherdthank you - yes, we will do our best to make it work before the deadline

Martin Grigorov (07:33:38) (in thread): > let me find an article explaining how to do it!

Martin Grigorov (07:34:13) (in thread): > https://blog.bioconductor.org/posts/2023-07-14-linux-arm64-github-actions/ - Attachment (Bioconductor community blog): Bioconductor community blog - Testing Packages on Linux ARM64 with GitHub Actions > How to use GitHub Actions to systematically build and test a Bioconductor package on Linux ARM64 architecture.

Martin Grigorov (07:35:12) (in thread): > You will have to replaceruns-on: ubuntu-22.04withruns-on: macos-14

Martin Grigorov (07:37:24) (in thread): > and you will have to simplify the example workflow by dropping the usage of Docker. Just run theR CMDcommands directly

Martin Grigorov (07:38:54) (in thread): > i.e. dropuses: addnab/docker-run-action@v3and itswith: .... For each step leavenameandrun

Matteo Tiberti (08:37:57) (in thread): > amazing Martin, thank you!

2024-04-04

Alexandru Mizeranschi (09:38:06): > @Alexandru Mizeranschi has joined the channel

2024-04-09

Aaron Lun (03:05:18) (in thread): > it appears that some process is still deleting my (i.e., gypum’s) cache on the build machines; this time it’s the mac builder for iSEE (https://bioconductor.org/checkResults/devel/bioc-LATEST/iSEE/lconway-buildsrc.html) and the linux builder for scuttle (https://bioconductor.org/checkResults/devel/bioc-LATEST/scuttle/nebbiolo1-buildsrc.html). > > To give some more background: the error message should only appear if (i) the data was downloaded and cached, (ii) aCOMPLETEmarker was added to the cache to indicate a successful download, but (iii) the data was deleted without also deleting theCOMPLETEmarker. If the download failed, the build should immediately stop with a error message (different from what is reported by the build system), the marker would not be added and the next build would attempt to redownload it. If the entire cache was deleted, the marker would not be present and the data would be re-downloaded on the next build. The error message emitted by the build system indicates that my functions believe that the download was successful and thus attempts to progress to the next step of using the downloaded files - which aren’t there, hence the HDF5 read errors. > > These sporadic failures are concerning for any package that depends ongypsum(includingscRNAseqand all of its reverse dependencies). Does anyone know what’s happening here?

Aaron Lun (03:17:01) (in thread): > For debugging purposes, thescuttlefailure should be reproducible with > > library(scRNAseq) > sce <- ZeiselBrainData() > > where the files are living at > > list.files(file.path(gypsum::cacheDirectory(), gypsum:::BUCKET_CACHE_NAME, "scRNAseq", "zeisel-brain-2015"), recursive=TRUE) > > And theCOMPLETEmarker is living at: > > list.files(file.path(gypsum::cacheDirectory(), "status", "scRNAseq", "zeisel-brain-2015"), recursive=TRUE) > > I usefilelockto lock cache writes across processes so I don’t think it’s some wacky race condition.

Aaron Lun (03:18:15) (in thread): > (of course, I don’t get any failure on my own machine, unless I go and manually delete the cached files without also deletingCOMPLETE.)

Andres Wokaty (10:00:09) (in thread): > I can look at this later today or tomorrow.

2024-04-12

Andres Wokaty (11:10:30) (in thread): > I wasn’t able to replicate the issue on lconway and nebbiolo1. I believe@Hervé Pagèsalso tried on nebbiolo1 and on his own machine. It seems like the builds are passing on those machines for those packages now. > > I am going to update R on the devel builders next week. Maybe that will create an opportunity for the error to reappear.

Aaron Lun (11:10:54) (in thread): > thanks. hm. well, fingers crossed again.

2024-04-17

Chenyue Lu (11:00:07): > @Chenyue Lu has joined the channel

2024-04-18

Aaron Lun (01:38:19) (in thread): > looks like our wish was granted, scuttle failed on nebbiolo (https://bioconductor.org/checkResults/devel/bioc-LATEST/scuttle/nebbiolo1-buildsrc.html) for this reason. > > Interestingly the same package also failed on kunpeng (https://bioconductor.org/checkResults/devel/bioc-LATEST/scuttle/kunpeng2-buildsrc.html) but because AnnotationHub’s Sqlite file was corrupted, presumably due to the same cache issues. > > Perhaps it would be prudent to not delete the cache when reinstalling R. There’s no real reason that the cache needs to be cleared; certainly users won’t be deleting their caches when updating R, so each package should already be robust to old caches if they exist.

Andres Wokaty (13:51:55) (in thread): > I deleted the cache when I installed R on nebbiolo1. I don’t understand why, for nebbiolo1, it isn’t the same behavior as a completely new install if there is no R cache. When I tried to reproduce the failure, it appears to be already resolved by the 2nd of run of the build, which had already happened and when I tried to reproduce it on nebbiolo1. I do still have to delete the caches for ExperiementHub and AnnotationHub a few times during a cycle, but maybe I don’t need to remove the cache for other packages. I don’t plan to remove the cache if I install R on devel again before the release.@Martin GrigorovCan you look at the AnnotationHub issue on kunpeng2?

Aaron Lun (13:54:41) (in thread): > hm. if it only fails on the first time, this suggests that there’s something wrong with my instantiation of the cache directory. but i did adddir.create(dirname(destination), recursive=TRUE, showWarnings=FALSE)in my code, so it should have been fine

Martin Grigorov (13:54:43) (in thread): > Today I installed R beta from Apr 15th.Should I delete caches ?

Andres Wokaty (14:02:55) (in thread): > @Martin GrigorovI am not very familiar with thedatabase disk image is malformederror. If you remove theAnnotationHubcache, can you reinstate it inRwith > > library(AnnotationHub) > ah <- AnnotationHub() >

Andres Wokaty (14:11:15) (in thread): > I will try to reproduce it with the BBS container and report back the result.

Aaron Lun (14:22:38) (in thread): > WOAH. I just flushed my own cache multiple times and i can sporadically (about 10% of the time) reproduce theError in H5Fopen(fpath) : HDF5. File accessibility. Unable to open file.error. It seems thataws.s3::save_objectdoes not throw an error message upon failed download.

Aaron Lun (14:31:05) (in thread): > looking at the code ofaws.s3::save_object, it seems they just ignore the status code froms3HTTP… great.

Aaron Lun (14:31:39) (in thread): > sigh … well, this is going to take a while to fix. suggest leaving the gypsum caches alone for a while so that we get stable builds at least.

2024-04-19

Aaron Lun (12:27:22) (in thread): > alright, swithced topaws.storagefor S3 interactions in the latest gypsum. note that this doesn’t really solve the underlying problem (i.e., why do the requests fail?) but at least we get more accurate error messages and we aren’t left with an invalid cache.

2024-04-25

Mercedes Guerrero (05:02:39): > @Mercedes Guerrero has joined the channel

2024-04-26

Clemens Kohl (04:56:07) (in thread): > Hi Jen, would it be possible to also install pytorch on kunpeng2 ? It has the same error. Thanks a lot in any case!

Alan Murphy (05:07:31) (in thread): > Hey@Martin Grigorov, apologies but this error is cropping up again on kunpeng2: > > Packages suggested but not available: > 'SNPlocs.Hsapiens.dbSNP144.GRCh38', > 'SNPlocs.Hsapiens.dbSNP155.GRCh38' >

Alan Murphy (05:09:23) (in thread): > Also happened on the long tests on Palomino3: > > Packages required but not available: > 'BSgenome', 'VariantAnnotation', 'rtracklayer' > > Packages suggested but not available: > 'SNPlocs.Hsapiens.dbSNP144.GRCh37', > 'SNPlocs.Hsapiens.dbSNP144.GRCh38', > 'SNPlocs.Hsapiens.dbSNP155.GRCh37', > 'SNPlocs.Hsapiens.dbSNP155.GRCh38', > 'BSgenome.Hsapiens.1000genomes.hs37d5', > 'BSgenome.Hsapiens.NCBI.GRCh38', 'GenomicFiles' >

Martin Grigorov (05:12:47) (in thread): > I will re-check later today but I’m sure I installed the big packages (2x6GB) for 4.4 beta

Andres Wokaty (09:20:46) (in thread): > @Martin Grigorovkunpeng2 appears to be missing pytorch.

Xiaosai Yao (14:21:57) (in thread): > Hi@Andres WokatyI am experiencing a similar error with the epiregulon package on lconway > > Error: processing vignette 'multiome.mae.Rmd' failed with diagnostics: > HDF5. File accessibility. Unable to open file > > https://bioconductor.org/checkResults/devel/bioc-LATEST/epiregulon/lconway-buildsrc.htmlThe vignette failed because it downloads the data fromExperimentHub. Could you kindly advise on how to resolve with this error?

Andres Wokaty (14:23:44) (in thread): > I will look at it a little later today and get back to you.

Xiaosai Yao (14:30:58) (in thread): > Thank you!!

Andres Wokaty (16:56:14) (in thread): > @Xiaosai YaoI was not able to reproduce the error. Maybe it is coming fromscuttle. It suggestsscRNAseq, which was updated yesterday but after the build started on lconway. Let’s see if it clears on the next report.

Aaron Lun (16:57:35) (in thread): > i don’t see how scuttle is getting involved here

Andres Wokaty (16:59:11) (in thread): > scuttle also had the same error earlier?

Aaron Lun (17:01:33) (in thread): > yes but that part of epiregulon doesn’t use scuttle AFAICT. in fact, the only disk-interfacing operation ismae <- scMultiome::reprogramSeq(), which is using HDF5 files cached from ExperimentHub.

Aaron Lun (17:03:36) (in thread): > basically, it seems like all cached HDF5 files are having trouble on the BBS, regardless of their source (ExperimentHub or gypsum). And in fact, even EHub’s own sqlite files are having trouble, see the latest scuttle errorhere.

Aaron Lun (17:04:12) (in thread): > the common denominator seems to be the cache.

Andres Wokaty (17:04:49) (in thread): > Thanks for pointing that out. I didn’t look into the packages because I was running bothR CMD checkon epiregulon and iSEE.

Andres Wokaty (17:06:33) (in thread): > I haven’t altered the cache recently.

Xiaosai Yao (17:17:25) (in thread): > @Andres Wokatythank you for looking into this. this is epiregulon’s first build report. but if the error fromscuttlehas persisted over the past few days, it’s unlikely that epiregulon’s build error will go away tomorrow?

Andres Wokaty (17:25:05) (in thread): > Aaron said it may not be fromscuttleso let’s disregard that for now. > > I ranR CMD buildandR CMD checkonepiregulonon lconway manually and it passed without errors. So I thought maybe the error may go away on the next build or this error is intermittent.

Andres Wokaty (17:35:35) (in thread): > If these issues aren’t resolved by next week, I’ll look into them after the release.

Xiaosai Yao (17:52:56) (in thread): > Thank you@Andres Wokaty!

2024-04-27

Martin Grigorov (02:46:17) (in thread): > > biocbuild@kunpeng2 ~> python3 > Python 3.9.9 (main, Oct 18 2023, 18:17:13) > [GCC 10.3.1] on linux > Type "help", "copyright", "credits" or "license" for more information. > >>> import torch > >>> > biocbuild@kunpeng2 ~> R > > R version 4.4.0 beta (2024-04-15 r86425) -- "Puppy Cup" > Copyright (C) 2024 The R Foundation for Statistical Computing > Platform: aarch64-unknown-linux-gnu > > R is free software and comes with ABSOLUTELY NO WARRANTY. > You are welcome to redistribute it under certain conditions. > Type 'license()' or 'licence()' for distribution details. > > Natural language support but running in an English locale > > R is a collaborative project with many contributors. > Type 'contributors()' for more information and > 'citation()' on how to cite R or R packages in publications. > > Type 'demo()' for some demos, 'help()' for on-line help, or > 'help.start()' for an HTML browser interface to help. > Type 'q()' to quit R. > > [Previously saved workspace restored] > > > library(reticulate) > > reticulate::import('torch') > Error in py_module_import(module, convert = convert) : > ModuleNotFoundError: No module named 'torch' > Run `reticulate::py_last_error()` for details. > > >

Martin Grigorov (02:47:34) (in thread): > pytorch is installed on the system but it is not visible to reticulate. What exactly should I do ? I thought that the R packages themselves use Reticulate to install whatever dependencies they need

Martin Grigorov (02:53:30) (in thread): > > py_install("torch") > Using virtual environment '/home/biocbuild/.virtualenvs/r-reticulate' ... > + /home/biocbuild/.virtualenvs/r-reticulate/bin/python -m pip install --upgrade --no-user torch > Collecting torch > Downloading torch-2.3.0-cp39-cp39-manylinux2014_aarch64.whl.metadata (26 kB) > Collecting filelock (from torch) > Downloading filelock-3.13.4-py3-none-any.whl.metadata (2.8 kB) > Collecting typing-extensions>=4.8.0 (from torch) > Downloading typing_extensions-4.11.0-py3-none-any.whl.metadata (3.0 kB) > Collecting sympy (from torch) > Using cached sympy-1.12-py3-none-any.whl.metadata (12 kB) > Collecting networkx (from torch) > Using cached networkx-3.2.1-py3-none-any.whl.metadata (5.2 kB) > Collecting jinja2 (from torch) > Using cached Jinja2-3.1.3-py3-none-any.whl.metadata (3.3 kB) > Collecting fsspec (from torch) > Downloading fsspec-2024.3.1-py3-none-any.whl.metadata (6.8 kB) > Collecting MarkupSafe>=2.0 (from jinja2->torch) > Using cached MarkupSafe-2.1.5-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.metadata (3.0 kB) > Collecting mpmath>=0.19 (from sympy->torch) > Using cached mpmath-1.3.0-py3-none-any.whl.metadata (8.6 kB) > Downloading torch-2.3.0-cp39-cp39-manylinux2014_aarch64.whl (88.5 MB) > ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 88.5/88.5 MB 1.2 MB/s eta 0:00:00 > Downloading typing_extensions-4.11.0-py3-none-any.whl (34 kB) > Downloading filelock-3.13.4-py3-none-any.whl (11 kB) > Downloading fsspec-2024.3.1-py3-none-any.whl (171 kB) > ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 172.0/172.0 kB 6.1 MB/s eta 0:00:00 > Using cached Jinja2-3.1.3-py3-none-any.whl (133 kB) > Using cached networkx-3.2.1-py3-none-any.whl (1.6 MB) > Using cached sympy-1.12-py3-none-any.whl (5.7 MB) > Using cached MarkupSafe-2.1.5-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (26 kB) > Using cached mpmath-1.3.0-py3-none-any.whl (536 kB) > Installing collected packages: mpmath, typing-extensions, sympy, networkx, MarkupSafe, fsspec, filelock, jinja2, torch > Successfully installed MarkupSafe-2.1.5 filelock-3.13.4 fsspec-2024.3.1 jinja2-3.1.3 mpmath-1.3.0 networkx-3.2.1 sympy-1.12 torch-2.3.0 typing-extensions-4.11.0 > > reticulate::import('torch') > Module(torch) > > Installed it! > But as I said - I thought that this is something the Bioc packages do themselves … > Please let me know if I did it wrong !

Clemens Kohl (06:01:46) (in thread): > Thanks a lot Martin for installing torch! Indeed itshouldinstall everything automatically, but I have been trouble getting it to work consistently. > I have these lines in my DESCRIPTION file > > Config/reticulate: > list( > packages = list( > list(package = "numpy"), > list(package = "torch") > ) > ) > > and additionally configure the environment according to the reticulate documentation inR/import_packages.R: > > # Load python packages > .onLoad <- function(libname, pkgname) { > reticulate::configure_environment(pkgname) > } > > On the machines I tested it, it automatically installs torch (although the whole thing is also a bit more complicated, because conda calls it pytorch, pip installs it via torch). But from other packages I know it doesnt always install all dependencies, e.g. I could never get umap-learn to be installed reliably. If there is someone to talk to with more experience with managing python deps, I would be happy to get some tips! I also tried basilisk a long time ago, but that was similarly error prone and its setup process almost always led to time out during the build processes on the Bioconductor servers.

Martin Grigorov (06:05:45) (in thread): > I am not the best person to give advices on this subject but let me know if you need more help with kunpeng2 !

Clemens Kohl (06:06:05) (in thread): > Thanks again, will do!

2024-04-28

Xiaosai Yao (14:16:42) (in thread): > the build reports are now fine. Thank you@Andres Wokaty

2024-04-29

Alan Murphy (09:22:59) (in thread): > Hey@Martin Grigorov, just as an update on kunpeng2, it is now showing the following: > > Package suggested but not available: 'SNPlocs.Hsapiens.dbSNP155.GRCh38' >

Martin Grigorov (09:23:16) (in thread): > duh

Martin Grigorov (09:24:27) (in thread): > is this package somehow related to AnnotationHub package ?

Alan Murphy (09:26:28) (in thread): > Not directly, it uses these database packages to run checks on the outputs of genetic studies (GWAS)

Martin Grigorov (09:30:34) (in thread): > this is my mistake … > You asked me about these missing packages 3 days ago. I started their download/install in a terminal tab but then forgot about them and stopped my dev machine … > Now I am re-installing them but in a tmux session so it won’t fail again. > Sorry!

Alan Murphy (09:33:52) (in thread): > No worries at all! Thanks for this!!

Martin Grigorov (09:38:40) (in thread): > > SNPlocs.Hsapiens.dbSNP155.GRCh37 # needed by MungeSumstats > SNPlocs.Hsapiens.dbSNP144.GRCh38 # needed by MungeSumstats > SNPlocs.Hsapiens.dbSNP155.GRCh38 # needed by MungeSumstats > > those are the packages needed by MungeSumstats, right ?

Alan Murphy (09:49:07) (in thread): > These 6 are all necessary: > > SNPlocs.Hsapiens.dbSNP144.GRCh37 > SNPlocs.Hsapiens.dbSNP144.GRCh38 > SNPlocs.Hsapiens.dbSNP155.GRCh37 > SNPlocs.Hsapiens.dbSNP155.GRCh38 > BSgenome.Hsapiens.1000genomes.hs37d5 > BSgenome.Hsapiens.NCBI.GRCh38 >

2024-05-01

Lori Shepherd (15:30:25): > Bioconductor Core Team is pleased to release Bioc3.19! Thank you to all developers and community members for contributing to the project. The full release announcement can be found at:https://bioconductor.org/news/bioc_3_19_release/

2024-05-04

Henrik Bengtsson (01:35:17): > The <https://bioconductor.org/packages/3.19/workflows/src/contrib/PACKAGES.gz> file is not up-to-date; it comprise old Bioc 3.19 packages; > > > db.gz <- read.dcf(gzcon(url("[https://bioconductor.org/packages/3.19/workflows/src/contrib/PACKAGES.gz](https://bioconductor.org/packages/3.19/workflows/src/contrib/PACKAGES.gz)"))) > > head(db.gz[,1:2]) > Package Version > [1,] "annotation" "1.27.0" > [2,] "arrays" "1.29.0" > [3,] "BiocMetaWorkflow" "1.25.0" > [4,] "BP4RNAseq" "1.13.0" > [5,] "CAGEWorkflow" "1.19.0" > [6,] "chipseqDB" "1.27.0" > > https://bioconductor.org/packages/3.19/workflows/src/contrib/PACKAGESis okay; > > > db <- read.dcf(url("[https://bioconductor.org/packages/3.19/workflows/src/contrib/PACKAGES](https://bioconductor.org/packages/3.19/workflows/src/contrib/PACKAGES)")) > > head(db[,1:2]) > Package Version > [1,] "annotation" "1.28.0" > [2,] "arrays" "1.30.0" > [3,] "BiocMetaWorkflow" "1.26.0" > [4,] "BP4RNAseq" "1.14.0" > [5,] "CAGEWorkflow" "1.20.0" > [6,] "chipseqDB" "1.28.0" >

Henrik Bengtsson (03:57:38) (in thread): > Same problem with <https://bioconductor.org/packages/3.19/workflows/src/contrib/PACKAGES.rds>; > > > download.file("[https://bioconductor.org/packages/3.19/workflows/src/contrib/PACKAGES.rds](https://bioconductor.org/packages/3.19/workflows/src/contrib/PACKAGES.rds)", destfile = "PACKAGES.rds") > trying URL '[https://bioconductor.org/packages/3.19/workflows/src/contrib/PACKAGES.rds](https://bioconductor.org/packages/3.19/workflows/src/contrib/PACKAGES.rds)' > Content type 'unknown' length 3836 bytes > ================================================== > downloaded 3836 bytes > > > db.rds <- readRDS("PACKAGES.rds") > > head(db.rds[,1:2]) > Package Version > annotation "annotation" "1.27.0" > arrays "arrays" "1.29.0" > BiocMetaWorkflow "BiocMetaWorkflow" "1.25.0" > BP4RNAseq "BP4RNAseq" "1.13.0" > CAGEWorkflow "CAGEWorkflow" "1.19.0" > chipseqDB "chipseqDB" "1.27.0" >

Lluís Revilla (04:34:54) (in thread): > A propos of this, are this files in Bioconductor written with tools::write_PACKAGES/update_PACKAGES? I thought it would ensure all three files would be up to date and synchronized.

Lori Shepherd (08:57:55) (in thread): > I think this has to do with our CloudFront caching. We need to set a reset after builds propagate

Lori Shepherd (09:11:19) (in thread): > I just did a manual reset and it should work now. I have a task for monday to set this up to do daily auto after the builds propagate for the day. We apologize for the inconvenience and confusion

Henrik Bengtsson (13:27:17) (in thread): > Wow, that was quick. I’m confirming I’m now seeing the proper versions; > > > db <- as.data.frame(utils::available.packages()) > > head(subset(db, grepl("workflows", Repository))[, 1:2]) > Package Version > annotation annotation 1.28.0 > arrays arrays 1.30.0 > BiocMetaWorkflow BiocMetaWorkflow 1.26.0 > BP4RNAseq BP4RNAseq 1.14.0 > CAGEWorkflow CAGEWorkflow 1.20.0 > chipseqDB chipseqDB 1.28.0 > > This bug manifested itself in Bioc workflow packages (https://bioconductor.org/packages/3.19/workflows) failed to install, because there was no tarball with those version available to download. So, it’s not that someone installed an older version - it just gave an error.

Henrik Bengtsson (17:18:58): > The following Bioconductor 3.19 packages depend on CRAN packages no longer on CRAN: > > MetaScope: depends onqlcMatrixarchived on 2023-11-29 > PiandRandomWalkRestartMH: depend ondnetarchived on 2024-01-30 > VarfromPDB[deprecated] andnanotatoR: depend onXML2Rarchived on 2024-03-24 > TissueEnrich: depends onensurerarchived on 2024-04-12 > > “Challenging” question: Given that these are broken, and were broken well() before the Bioc 3.19 release on 2024-05-01, should they even have made it into Bioc 3.19 in the first place? > > () The exception being TissueEnrich, which broke 19 days before the release.

Henrik Bengtsson (17:22:58) (in thread): > Also, the build system reports OK for both ‘BUILD’ and ‘INSTALL’ for ‘MetaScore’, and only detects the ERROR in ‘CHECK’https://bioconductor.org/checkResults/release/data-experiment-LATEST/MetaScope/). > > Does this mean the Bioc build system holds on the installed CRAN packages that have since been archived/removed? If so, it’s odd that this only happens for the MetaScore package and not for the other packages mentioned above.

Lori Shepherd (18:42:33) (in thread): > We have to manually remove CRAN packages that get removed. Normally we pick it up when we do a re-install of R. They still would have made/be included in the release regardless but the debate would be weather they should’ve been marked deprecated before the release with the intent of removal if not fixed in 3.20 rather than a whole cycle more.

Lori Shepherd (19:03:43) (in thread): > We try to give packages a fair bit of time to fix before we remove

Henrik Bengtsson (21:42:27) (in thread): > Okay, thanks for clarifying. > > Normally we pick it up when we do a re-install of R. > I’m not sure, but I thinkR CMD check --as-cranreports on dependencies that are archived CRAN packages. I haven’t check the code, but it might be possible to set an env var so that this is reported as a NOTE (WARNING?) also in the Bioc checks. > > So, if you refresh the Bioc builders package library to not hold archived CRAN packages, then ‘qclMatrix’ will vanish from there, meaning MetaScope should fail in the next build cycle, and then no longer be served via PACKAGES. Is that correct? > > … the debate would be weather they should’ve been marked deprecated before the release with the intent of removal if not fixed in 3.20 rather than a whole cycle more. > Yes, I think these should have been automatically deprecated, because they will hang around for another 12 months now. If MetaScope is not fixed, it means Bioconductor will listed this package for 12+5=17 months after it first broke. In contrast, CRAN would kick that package out in 2 weeks … actually, they would auto-archive it immediately when qlcMatrix was archived.

2024-05-05

Kasper D. Hansen (03:10:49) (in thread): > I think we should be substantially more aggressive about removal when a package fails because the dependency gets removed.Not 2 weeks but much faster than 17 months

Lluís Revilla (17:50:12) (in thread): > I agree with Henrik and Kasper, why have 6 months release cycle if at the end it doesn’t work to pick these up? It is relatively easy to check if a package in Bioconductor depends on a package removed from CRAN.

Lori Shepherd (20:50:21) (in thread): > So this to me seems directly in conflict then with the push to use CRAN Haven to lessen reports of removed CRAN packages… We can’t have both directions pushed.
> > CRAN receives a lot of negative comments in how they remove packages; we try to walk a balance with regards to how long a packages is failing on our system (actually picked up on our system) and how many times a maintainer is contacted. > > Yes there could/should be an investigation in how to better pick up these changes rather than waiting for an R update….

2024-05-06

Henrik Bengtsson (03:52:41) (in thread): > Re CRANhaven: I wouldn’t say there’s a conflict; The two main purposes of CRANhaven are: (1) to give a buffer to end users when a package is suddenly archived on CRAN, and (2) to mitigate the impact of packages that aretemporarilyarchived on CRAN. Bioconductor users and developers are some who would benefit from this CRANhaven cushion. > > If CRAN would let the public know at the same time they give the maintainer a two-week notice that a package will be archived unless fixed, then the first purpose would have much less value. But without that notice to the public (read end-users and other developers), then the archival of a package is very abrupt and disruptive. There is not way to plan for it. > > For the second purpose, CRANhaven is set to hold on to archived packages for 35 days, which means that 50% of the packages that do indeed get fixed and return to CRAN will do so within this time span. The cushion is currently 35 days, but that is not set in stone. > > FWIW, in the future R-Universe might support multiple repositories under the same umbrella, which means we could imagine providing CRANhaven repositories with differently life lengths, e.g. 14 days, 1 month, 3 months, 6 months. Until that’s available, I/we are happy to fine tune it to whatever would be useful for Bioconductor.

Lluís Revilla (06:57:35) (in thread): > The way I see it is that identifying when it happens would allow to measure the impact of CRANhaven and help the developers (and users). > 1. A package is removed from CRAN (I don’t know if CRAN sends emails to maintainers from other repositories) > 2. CRANhaven gives extra time to the developer > 3. If CRAN package is not back in X time, the developer could be notified that it is unlikely one of their dependencies will be back and should adapt their package. > This would help the developers by giving more time to adapt and Bioconductor would not release a package that depends on a missing package for so long.

2024-05-15

Karrar (09:05:19): > @Karrar has joined the channel

2024-05-23

Lori Shepherd (14:20:42): > Due to unforeseen circumstances, our daily builders and new submission builders are currently offline. We are sorry for the inconvenience and will restore as soon as possible

2024-07-13

koki (09:10:44): > Hello, BioC Core Team. > My packagescTensorhas build error on the Linux/macOS x86_64 machines in BioC 3.19 and the Linux machine in BioC 3.20.https://bioconductor.org/checkResults/release/bioc-LATEST/scTensor/https://bioconductor.org/checkResults/3.20/bioc-LATEST/scTensor/The error is related to this part but I cannot reproduce the same error in my local machine. > > library("AnnotationHub") > ah <- AnnotationHub() > hs <- query(ah, c("OrgDb", "Homo sapiens"))[[1]] > > Would a little time resolve this error? > Could you check the detail?

2024-07-15

Lori Shepherd (07:34:22): > this should be resolved

2024-07-16

Hiru (06:56:33): > @Hiru has joined the channel

2024-07-21

koki (00:26:02) (in thread): > Sorry, you mean I should fix the vignette ofscTensor? > Or do you mean that you plan to look into the server environment because it is odd thatorg.Hs.eg.dbcannot be retrieved from theAnnotationHub?

Lori Shepherd (12:39:58) (in thread): > It should have been resolved via annotationhub as I reset the resource. if it reoccurred then there might be an issue in the code. Is it still occurring?

koki (18:49:47) (in thread): > Thanks, I will monitor daily build for a while.

2024-07-28

koki (06:43:29) (in thread): > It seems that the same build error occured in the July 24th update only on the macOS machine.https://bioconductor.org/checkResults/3.19/bioc-LATEST/scTensor/

2024-07-29

Lori Shepherd (07:48:04) (in thread): > thanks we’ll investigate further on the builder side. question: in your code do you set any timeout limits or anything?

koki (09:20:41) (in thread): > > timeout limits > I think no. > I just run the codes with ordinary R-markdown options. > > cf.https://github.com/rikenbit/scTensor/blob/bfe66bccf230d9786dc277bd00d01f57f724bc33/vignettes/scTensor_1_Data_format_ID_Conversion.Rmd#L41https://www.bioconductor.org/packages/release/bioc/vignettes/scTensor/inst/doc/scTensor_1_Data_format_ID_Conversion.R

Lori Shepherd (09:21:21) (in thread): > ok. thank you

2024-08-06

Xiaosai Yao (23:27:56): > Hi core team, I have a recentbuild errorwith scMultiome in BioC3.19, likely due to a change in DelayArray. I could not reproduce the error in BioC3.19 but obtained the exact error in BioC3.20. Could you please verify that scMultiome was in fact built from BioC3.19? Thank you!

2024-08-07

Andres Wokaty (10:59:38) (in thread): > It is built from 3.19. A few weeks ago R was updated on the build machines to 4.4.1, both release and devel. I would check that your 3.19 is running on R 4.4.1.

Xiaosai Yao (12:33:20) (in thread): > My R version is 4.4.1

Andres Wokaty (12:39:28) (in thread): > Could you share your sessionInfo()?

Xiaosai Yao (13:44:45) (in thread): > > sessionInfo() > R version 4.4.1 (2024-06-14) > Platform: x86_64-apple-darwin20 > Running under: macOS Sonoma 14.5 > > Matrix products: default > BLAS: /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib > LAPACK: /Library/Frameworks/R.framework/Versions/4.4-x86_64/Resources/lib/libRlapack.dylib; LAPACK version 3.12.0 > > locale: > [1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8 > > time zone: America/Los_Angeles > tzcode source: internal > > attached base packages: > [1] stats graphics grDevices utils datasets methods base > > loaded via a namespace (and not attached): > [1] compiler_4.4.1 tools_4.4.1 rstudioapi_0.16.0 >

2024-08-08

Hervé Pagès (02:57:20) (in thread): > @Andres WokatyLooks like nebbiolo1 used the devel versions of some packages e.g.DelayedArray(0.31.11) andSparseArray(1.5.31) for the latest BioC 3.19 data-experiment builds:https://bioconductor.org/checkResults/3.19/data-experiment-LATEST/nebbiolo1-R-instpkgs.htmlNot sure how this could happen:thinking_face:

Andres Wokaty (12:20:32) (in thread): > When I look at nebbiolo1 now, I only see the release version, although I see what you mean and it looks like this happened for the annotation builds but not for the long builds over the weekend.@Xiaosai YaoWhen I tried to build and check scMultiome manually, it used the release version of DelayedArray, so it should use the correct version today.

Xiaosai Yao (18:08:22) (in thread): > Thanks for looking into this@Hervé Pagèsand@Andres Wokaty. The build is fine today. The error seems somewhat sporadic. I will alert you if the error comes up again

2024-08-09

Charlotte Soneson (06:52:50): > Three of our packages using python code (sketchR,orthosandvelociraptor, I also saw a couple of other packages with the same issue) currently fail on the Windows release builder (palomino7) with similar error message, and I’m trying to diagnose what the issue might be. For example, I see (this one fromsketchR): > > CondaVerificationError: The package for scanpy located at C:\Users\BIOCBU~1\AppData\Local\R\cache\R\basilisk\116~1.0\0\pkgs\scanpy-1.7.2-pyhdfd78af_0 > appears to be corrupted. The path 'site-packages/scanpy/tests/_images/embedding-missing-values/test_missing_values_categorical[spatial-na_color.black_tup-na_in_legend.True-legend.on_right-groups.all].png' > specified in the package manifest cannot be found. > > ClobberError: This transaction has incompatible packages due to a shared path. > packages: conda-forge/win-64::git-2.43.0-h57928b3_1, conda-forge/win-64::git-2.43.0-h57928b3_1 > path: 'menu/menu-windows.json' > > I’m wondering if it’s possible that there could be corrupted files in the cache (if so, perhaps aconda clean --allmight help)? It is also true thatscanpy1.7.2 (the version that is used here) is in fact available from two channels (biocondaandconda-forge) and I wonder if that could be an issue if different Bioc packages have different channel priorities (however, as far as I can remember this is the first time I see this particular error, I would have somehow expected this to have happened before in that case, since the packages have been around for a while).

Andres Wokaty (17:31:40) (in thread): > I triedconda clean --allonpalomino7today but got an error that it can’t find anotherscanpyfile, although that file appears to exist in that path. Should I try removingscanpymanually? I’m not knowledgeable aboutconda. Maybe we could set up a time to look at palomino7 together next week when you’re available? - File (PNG): Screenshot from 2024-08-09 16-31-52.png

koki (20:46:23) (in thread): > Hi, the build error on macOS machine has fixed. Thank you.https://bioconductor.org/checkResults/3.19/bioc-LATEST/scTensor/

Hervé Pagès (23:59:55) (in thread): > Maybe try to remove the entire basilisk cache altogether?

2024-08-10

Charlotte Soneson (01:38:14) (in thread): > Yes, either removingscanpyor clearing the cache might be a solution. I’m off on vacation the coming week but happy to look into it together when I’m back if the problem persists! Thanks!

2024-08-12

Andres Wokaty (17:17:40) (in thread): > I tried removing onlyscanpybut the persisted. The same version is used onpalomino8. I have removed thebasiliskcache, so we should check if the problem is resolved on the Friday report.

2024-08-13

Alan Murphy (09:40:16): > Hey! Just in case you aren’t aware, the long tests in the devel branch don’t seem to have updated for a number of weeks now:https://bioconductor.org/checkResults/3.20/bioc-longtests-LATEST/

Xiaosai Yao (18:40:54) (in thread): > I received the same error today@Andres Wokatyhttps://bioconductor.org/checkResults/3.19/data-experiment-LATEST/scMultiome/nebbiolo1-checksrc.html

Andres Wokaty (18:59:58) (in thread): > Thanks for letting me know. I will take a look.

Andres Wokaty (22:05:42) (in thread): > Thanks for bringing that to our attention. I’ll take a look.

Andres Wokaty (22:08:30) (in thread): > I’ve reinstalled R on nebbiolo1. When I checked, it was using the wrong version of Bioconductor and a few packages seemed affected. If the problem continues to happen, I’ll do more troubleshooting. Thanks for your patience.

Xiaosai Yao (22:09:12) (in thread): > Thanks for troubleshooting!

2024-08-16

Charlotte Soneson (17:55:47) (in thread): > Looks like these are still failing today:disappointed:

2024-08-19

Rema Gesaka (09:37:15): > @Rema Gesaka has joined the channel

2024-08-20

Andres Wokaty (11:11:38) (in thread): > @Charlotte SonesonWould you like to try troubleshooting together? I have time before 11AM ET Thursday and Friday.scanpyseems to have the same problem everytime it gets reinstalled after removing it or the entirebasiliskcache. I noticed there also seems to be some issues indevelnow too.

Charlotte Soneson (11:13:17) (in thread): > Sure! I can do 9:30-11:00 ET on Friday, would that work?

Andres Wokaty (11:14:04) (in thread): > Sounds good. We can huddle or whatever you prefer.

Andres Wokaty (11:14:27) (in thread): > I will be available at 9:30.

Charlotte Soneson (11:15:14) (in thread): > I’ll send you an invite by email (somehow I never manage to join huddles, may be a firewall issue:woman-shrugging:)

2024-09-06

Lori Shepherd (07:12:19): > The 3.20 release schedule has been announced. Please be aware of important deadlines:https://bioconductor.org/developers/release-schedule/

2024-09-10

Alex Qin (03:46:13): > @Alex Qin has joined the channel

Philippa Doherty (09:49:00): > @Philippa Doherty has joined the channel

2024-09-13

Gobi Dasu (18:19:26): > @Gobi Dasu has joined the channel

2024-09-19

Jonathan Wang (14:42:51): > @Jonathan Wang has joined the channel

2024-09-20

Alan Murphy (12:12:42) (in thread): > Hey@Martin Grigorov, I think ‘SNPlocs.Hsapiens.dbSNP155.GRCh37’ (and I would guess the other packages stated above) need to be installedkunpeng2 based on the Bioc 3.20 checks. Just wondering too, is this still the best way to report such issues?

2024-09-21

Martin Grigorov (01:42:45) (in thread): > Hi@Alan Murphy!I’llcheck why they are missing soon! Ididn’tupdate R recently…Slack works well for me! But you can also reach me via email (martin.grigorov @ gmail) and/or GitHub (martin-g)

2024-09-23

Alan Murphy (02:43:42) (in thread): > Thanks Martin!!

Martin Grigorov (02:47:08) (in thread): > indeed the package is missing ! Installing it/them !

Martin Grigorov (09:49:05) (in thread): > > > BiocManager::install(c("SNPlocs.Hsapiens.dbSNP155.GRCh37", "SNPlocs.Hsapiens.dbSNP155.GRCh38", "SNPlocs.Hsapiens.dbSNP144.GRCh37", "SNPlocs.Hsapiens.dbSNP144.GRCh38", "BSgenome.Hsapiens.1000genomes.hs37d5", "BSgenome.Hsapiens.NCBI.GRCh38")) > Bioconductor version 3.20 (BiocManager 1.30.25), R 4.4.1 (2024-06-14) > Old packages: 'geepack', 'rworkflows', 'shinyWidgets', 'tseries', 'XLConnect' > Update all/some/none? [a/s/n]: n > Warning message: > package(s) not installed when version(s) same as or greater than current; use > `force = TRUE` to re-install: 'SNPlocs.Hsapiens.dbSNP155.GRCh37' > 'SNPlocs.Hsapiens.dbSNP155.GRCh38' 'SNPlocs.Hsapiens.dbSNP144.GRCh37' > 'SNPlocs.Hsapiens.dbSNP144.GRCh38' 'BSgenome.Hsapiens.1000genomes.hs37d5' > 'BSgenome.Hsapiens.NCBI.GRCh38' > > all should be good now!

2024-09-30

Aaron Lun (01:45:44): > TileDBArrayis failing inrelease mac x86(but notdevel) due to an illegal opcode error . This suggests that an ARM version of the underlyingtiledbpackage was accidentally installed on the mac x86 release build machine.

Andres Wokaty (10:47:13) (in thread): > The version looks correct: > > merida1:~ biocbuild$ otool -L /Library/Frameworks/R.framework/Versions/4.4-x86_64/Resources/library/tiledb/libs/tiledb.so > /Library/Frameworks/R.framework/Versions/4.4-x86_64/Resources/library/tiledb/libs/tiledb.so: > tiledb.so (compatibility version 0.0.0, current version 0.0.0) > @rpath/libtiledb.dylib (compatibility version 0.0.0, current version 0.0.0) > /Library/Frameworks/R.framework/Versions/4.4-x86_64/Resources/lib/libR.dylib (compatibility version 4.4.0, current version 4.4.1) > /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 1775.118.101) > /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 905.6.0) > /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1292.100.5) > > But something else could be going on.

2024-10-01

Hervé Pagès (18:49:42) (in thread): > Right, atiledbinstallation from the wrong binary would have very likely preventedTileDBArrayfrom loading at the end of theR CMD INSTALL. More likely some memory corruption problems with the latesttiledb(the latest version, 0.30.0, was published on CRAN on Sep 11).

2024-10-05

Ochogwu Emmanuel (05:18:08): > @Ochogwu Emmanuel has joined the channel

Ebele Lynda Okolo (13:29:10): > @Ebele Lynda Okolo has joined the channel

2024-10-14

Clemens Kohl (06:11:19): > Hi, I recently moved my package APL to use basilisk in order to simplify the installation of python dependencies (mainly PyTorch & numpy for faster SVD). I tested it on the Bioconductor Docker containers and in a Debian VM and all seemed to work fine. The most recent builds on the Bioconductor servers however all fail, remarkably all with different errors (https://bioconductor.org/checkResults/devel/bioc-LATEST/APL/). Some I could tackle (e.g. on the Windows machine the specified torch version doesn’t seem to be available), but others seem to stem from missing libraries or even missing permissions to create the required environments. Could someone please advise me on how to best proceed or what changes I could make to resolve these errors. I am unfortunately not able to reproduce any of the errors. > Alternatively I could also roll-back the basilisk changes, but it seems it then requires the Bioc-team to manually install torch & numpy each release.

Martin Grigorov (06:24:51) (in thread): > atkunpeng2(Linux ARM64) it fails with versionGLIBCXX_3.4.32’ not found. Islibprotobuf.so.25.3.0built byR CMD build …`or it is downloaded prebuilt ?

Martin Grigorov (06:25:35) (in thread): > It looks to me that it comes prebuilt with some newer version of the GNU compilers (e.g. ver. 13 or 14)

Martin Grigorov (06:26:46) (in thread): > I could install gcc/g++ 14.x on the system if you think it is a good idea

Clemens Kohl (09:15:38) (in thread): > Hi Martin, thanks for the hint! Wouldn’t/shouldn’t Conda pull in the required gcc version when installing PyTorch? While it seems it could be a temporary fix, it seems to me it would go a bit against the idea of using basilisk to create a Conda environment that does not mess up the users local configuration.

Martin Grigorov (09:17:44) (in thread): > Conda only downloads the pre-built packages. It does not build them from source when they are used as dependencies

Martin Grigorov (09:18:15) (in thread): > Conda-forge CI builds the packages and uses the required gcc versions

Clemens Kohl (09:21:36) (in thread): > I see, thanks. Should I try to specify the channel to Conda-forge then via basilisk (although it seems to be the default)

2024-10-16

Matteo Tiberti (04:19:00): > hi everyone, ourMoonlight2R packagefails build because of 403 errors (Forbidden) or sometimes a timeout error when trying to access external resources, in particular NCBI. This happens for 3.20 builders, but not for 3.19 builders at the moment. It’s not the first time we see this, and could never replicate it locally. I was wondering if you have any suggestion on how to mitigate this problem, aside from not executing stuff, (e.g. some examples or vignettes or tests) or maybe repeating different attempts with a delay, both of which aren’t great solutions

Martin Grigorov (04:28:59) (in thread): > is the downloaded data cached or is it downloaded for everybuild?

Martin Grigorov (04:30:42) (in thread): > many packages save such data in ~/.cache/R/****. We can download it once there and reuse it

Matteo Tiberti (04:32:47) (in thread): > it is downloaded from every build - I think the assumption is that every build is independent, and so they should each download the data, isn’t that the case?

Lori Shepherd (06:20:35) (in thread): > Cachingof external resources is currently always advised because of the frequency of access. We will flush the (biocfilecace and hubs) cache occasionally to check endpoints and if you use experiment or annotation hub I’m working on a validation script to be run separately for that

Matteo Tiberti (11:11:00) (in thread): > ok, I’ll look into caching. Thank you very much!

Lori Shepherd (11:11:50) (in thread): > yep BiocFileCache might be helpful

Matteo Tiberti (11:12:20) (in thread): > Cool! I’m on it:+1:

Clemens Kohl (16:46:08) (in thread): > HI@Martin Grigorov, sorry to keep bothering you about this, but I looked a bit more into it as I am not extremely familiar with the details of conda & basilisk (as you might have guessed) and I am still at a loss on how to best solve this issue. I think therefore the easiest solution for the upcoming release would be if I revert the changes and try to find a suitable R native replacement SVD or solve the basilisk issue for the next release. However, the revert would then require you to install numpy & pytorch on the servers (alternatively, if it is installed in the “r-reticulate” conda environment provided by reticulate this is sufficient too). Would that be possible? Thanks a lot in any case.

2024-10-17

Martin Grigorov (02:12:53) (in thread): > Hi@Clemens Kohl! Usually R packages use reticulate/basilisk to install Python dependencies. > The issue with libprotobuf.so is that it requires a rather new libstdc++ that is not available on the builder. Atkunpeng2we have GNU compiler suite v.12.3. It provides GLIBCXX_3.4.30. > Let me install v.14.x. This should solve the issue on the builder! But your users may face problems later if they use a Linux distro with older libstdc++.so, e.g. Ubuntu 22.04 will fail too

Martin Grigorov (02:13:49) (in thread): > Can you use an older version of libprotobuf that provides the needed functionalities by your software but is built with older libstdc++ ? This would be another solution too

Clemens Kohl (17:36:40) (in thread): > Thanks for the help! Which libstdc++ version compatibilty should I best aim for? I am currently trying to find out which version comes with a compatible libprotobuf, but unfortunately I can not find anywhere an overview of the dependencies of pytorch, this might take some time/experimentation. Interestingly, I tried installing APL on a Ubuntu 20.04 machine at work in a docker (same OS) and there it worked out, not sure where the differences are. > I was using reticulate before the latest changes, but even though I set up everything according to the guide, reticulate only installed numpy, never pytorch at first load. That is the reason why I wanted to switch to basilisk to finally solve the issue, as previously you had to install numpy & pytorch manually. But what worries me about the build errors currently is that on each server I get completely different errors, which all seem difficult to solve without being able to try out different configurations.

Clemens Kohl (17:42:41) (in thread): > I saw that libstdcxx-ng and libprotobuf are both available from conda too. Would it be sufficient if I pulled in compatible versions through conda?

2024-10-18

Martin Grigorov (11:32:31) (in thread): > if libprotobuf is a transitive dependency then it will be harder for you to figure out the version… I will install GNU compilers 14.x onkunpeng2during the weekend !

2024-10-19

Martin Grigorov (03:00:48) (in thread): > > biocbuild@kunpeng2 ~/git> R CMD build APL (base) > * checking for file 'APL/DESCRIPTION' ... OK > * preparing 'APL': > * checking DESCRIPTION meta-information ... OK > * installing the package to build vignettes > * creating vignettes ... OK > * checking for LF line-endings in source and make files and shell scripts > * checking for empty or unneeded directories > * building 'APL_1.9.5.tar.gz' > > all is fine with > > $ gcc --version (base) > gcc (conda-forge gcc 14.2.0-1) 14.2.0 > > andlibstdc++.so.6.0.33!

Martin Grigorov (03:02:21) (in thread): > it should be green onkunpeng2in the next report !

Clemens Kohl (06:25:26) (in thread): > Thanks a lot Martin! I really appreciate the help! Do you happen to know who I could contact about the build errors on the two Ubuntu servers (teran2 and nebbiolo2) as well as the macOS machine?

Martin Grigorov (06:26:34) (in thread): > @Andres Wokaty^^

2024-10-22

Liyang Fei (00:10:56): > @Liyang Fei has joined the channel

Edward Zhao (03:13:51): > @Edward Zhao has joined the channel

Edward Zhao (03:16:23): > Hello all, wondering if anyone else has run into a problem with the R arrow package causing build errors on macOS. > > Error: processing vignette 'VisiumIO.Rmd' failed with diagnostics: > unable to load shared object '/Library/Frameworks/R.framework/Versions/4.4-x86_64/Resources/library/arrow/libs/arrow.so': > dlopen(/Library/Frameworks/R.framework/Versions/4.4-x86_64/Resources/library/arrow/libs/arrow.so, 0x0006): symbol not found in flat namespace (_ZSTD_compress) > --- failed re-building 'VisiumIO.Rmd' > > See links here: > * BayesSpace on BioC 3.20 fails:https://bioconductor.org/checkResults/devel/bioc-LATEST/BayesSpace/ > * VisiumIO on BioC 3.20 fails:https://bioconductor.org/checkResults/devel/bioc-LATEST/VisiumIO/ > * VisiumIO on BioC 3.19 is ok:https://bioconductor.org/checkResults/release/bioc-LATEST/VisiumIO/ > > * interestingly, it seems like BioC 3.19 uses a newer version of macOS to test than BioC 3.20 > The dependency failing here is a CRAN package and passes all of their checks. > > Any help here would be appreciated. Thanks!

Martin Grigorov (03:26:45) (in thread): > Iszstdan optional dependency ofarrow? Maybe you need to provide it

2024-10-23

Martin Grigorov (01:40:01) (in thread): > @Clemens Kohlhttps://bioconductor.org/checkResults/devel/bioc-LATEST/APL/kunpeng2-buildsrc.htmlnow fails with the same error as for Linux x86_64. MKL is Intel specific library and won’t work on Linux ARM64. I wonder why it didn’t fail when I executedR CMD build

Edward Zhao (02:59:21) (in thread): > are you saying I should add zstdr (the cran R package) as a dependency here? arrow does not seem to require or suggest this

Martin Grigorov (03:09:17) (in thread): > well, the error says that loading arrow.so fails due to missing_ZSTD_compresswhich looks to me to be provided by zstd

Edward Zhao (03:19:05) (in thread): > isn’t this something that needs to be installed on the build environment?

Martin Grigorov (03:30:52) (in thread): > maybe

Martin Grigorov (03:31:11) (in thread): > https://bioconductor.org/checkResults/devel/bioc-LATEST/BayesSpace/kunpeng2-install.html- the issue with openblas has been fixed !

Edward Zhao (03:31:24) (in thread): > ok thanks. i saw that one was affecting many packages

Martin Grigorov (03:31:29) (in thread): > it should be OK in the next build (2 days from now)

Martin Grigorov (03:32:41) (in thread): > https://github.com/search?q=repo%3ABioconductor%2FBBS%20zstd&type=codethere is nothing aboutzstdin the README/install files

Edward Zhao (03:33:00) (in thread): > this one with arrow is affecting packages that depend on arrow. other than BayesSpace, i see VisiumIO (https://bioconductor.org/packages/3.20/bioc/html/VisiumIO.html) and SpatialFeatureExperiment (https://bioconductor.org/checkResults/devel/bioc-LATEST/SpatialFeatureExperiment/) - Attachment (Bioconductor): VisiumIO (development version) > The package allows users to readily import spatial data obtained from either the 10X website or from the Space Ranger pipeline. Supported formats include tar.gz, h5, and mtx files. Multiple files can be imported at once with *List type of functions. The package represents data mainly as SpatialExperiment objects.

Edward Zhao (03:34:16) (in thread): > i think this is related to arrow dropping support for macOS 12https://github.com/apache/arrow/issues/44206

Edward Zhao (03:35:38) (in thread): > arrow passes CRAN checks but it seems like CRAN uses macOS 13

Martin Grigorov (03:38:10) (in thread): > can you use an older version of arrow ? the last one that supports osx 12 ?

Martin Grigorov (03:39:08) (in thread): > hm, Arrow 17 has been released on Jul 16th. The PR above is from last month. I think it is not yet released

Edward Zhao (03:56:12) (in thread): > ok. it’s hard for me to test this since i do not have a mac

Edward Zhao (03:57:11) (in thread): > what do you suggest I do at this point? arrow is used for a fairly minor part of our package. I can take it out possibly. But really this seems to not be a problem with my package (or even the arrow package).

Martin Grigorov (03:58:19) (in thread): > you can write to bioc-devel mailing list and ask zstd to be installed on the builder machine

Edward Zhao (03:58:27) (in thread): > ok

Edward Zhao (03:58:32) (in thread): > thanks i will do that

Edward Zhao (04:04:21) (in thread): > isbioc-devel@bioconductor.organdbioc-devel@r-project.orgthe same? which one should i send to

Martin Grigorov (05:12:09) (in thread): > bioc-devel@r-project.org- I see your message there

Edward Zhao (12:51:11) (in thread): > great, thanks

Hong Qin (17:46:18): > @Hong Qin has joined the channel

2024-10-24

Matteo Tiberti (05:25:52): > hi everyone, seems like our build errors for Moonlight2R related to downloads have solved themselves which is good, but we still have a couple of warnings on two out of three of the Linux servers, which we haven’t had before on the builders: > > Found the following significant warnings: > Warning: 'rgl.init' failed, running with 'rgl.useNULL = TRUE'. > See '/home/biocbuild/bbs-3.20-bioc/meat/Moonlight2R.Rcheck/00install.out' for details. > > I’ve seen this before on our local servers and it could be related to either missing OpenGL support and/or the fact that it is running on a headless server (not sure why it pops up on nebbiolo2 and not teran2 though). A workaround might be to set the system variableRGL_USE_NULLtoTRUEprior to building (as impliedin the docs). Any other suggestion?

2024-10-25

Clemens Kohl (08:48:18) (in thread): > Hi Martin, that is at least because I tried adding it to the dependencies for the other servers where it seems to be missing. I can remove it again (and will) but I was hoping to figure out how to solve it everywhere. I tried the BBS salt containers but no luck so far.

Clemens Kohl (08:51:34) (in thread): > I will not be able to work on it over the weekend, but does the Bioc team have any recommendations on how to proceed here? If I try installing on the BBS container it only complains about missing permissions to create the conda environment, so I am stuck there

Martin Grigorov (08:52:48) (in thread): > Hi Clemens! > I have no experience with the salt containers … > You want to solve it how ? MKL won’t work on non-Intel hardware

Martin Grigorov (08:54:06) (in thread): > for kunpeng2 it fails again with: > > /usr/lib64/libstdc++.so.6: version `CXXABI_1.3.15' not found > > this was my mistake. It should be OK in the next build

Martin Grigorov (08:55:14) (in thread): > do you installhttps://anaconda.org/conda-forge/mklwith reticulate/basilisk ?

Clemens Kohl (08:59:45) (in thread): > I understand that it won’t work on non-Intel hardware, but the Ubuntu servers showlibmkl_intel_lp64.so.2: cannot open shared object file: No such file or directory, so in lack of any way to reproduce the errors I attempted to also install mkl through condo forge (as you mentioned) to understand if that would solve the problem, but evidently it does not.

Martin Grigorov (09:00:33) (in thread): > how did you install it ?

Clemens Kohl (09:01:43) (in thread): > with basilisk

Clemens Kohl (09:03:24) (in thread): > > #' @importFrom basilisk BasiliskEnvironment > APL_env <- BasiliskEnvironment( > envname = "APL-env", > pkgname = "APL", > packages = c( > "numpy==1.26.4", > "pytorch==2.3.0", > "libstdcxx-ng==14.2.0", > "libprotobuf==4.25.3", > "mkl==2023.2.0" > ) > ) > > to be specific

Martin Grigorov (09:05:03) (in thread): > I just downloaded 2024.2.2 and the file is there: > > /tmp/mkl > ❯ find ./ -name "**libmkl_intel_lp64.so.2**" > ./lib/libmkl_intel_lp64.so.2 >

Martin Grigorov (09:06:00) (in thread): > “libstdcxx-ng==14.2.0”, > here is why you neededGLIBCXX_3.4.32:slightly_smiling_face:

Clemens Kohl (09:06:57) (in thread): > I added that line actually after our conversation, as (maybe misguidedly?) I thought this might provide GLIBCXX

Martin Grigorov (09:14:29) (in thread): > it might help! But it is provided by the builder now too

Clemens Kohl (11:01:01) (in thread): > > I just downloaded 2024.2.2 and the file is there: > > > /tmp/mkl > ❯ find ./ -name "**libmkl_intel_lp64.so.2**" > ./lib/libmkl_intel_lp64.so.2 > > I this folder also in LD_LIBRARY_PATH?

Martin Grigorov (11:09:12) (in thread): > Basilisk should make it available.But I am not R developer so there might be something more needed

2024-10-26

Andres Wokaty (10:00:00) (in thread): > Sorry for responding late and please disregard my message over the weekend. I looked atAPLonnebbiolo2.mlkdid exist inbasilisk. I was able to run the following with no issue: > > #' @importFrom basilisk BasiliskEnvironment > APL_env <- BasiliskEnvironment( > envname = "APL-env", > pkgname = "APL", > packages = c( > "numpy==1.26.4", > "pytorch==2.3.0", > "libstdcxx-ng==14.2.0", > "libprotobuf==4.25.3", > "mkl==2023.2.0" > ) > ) > > So I tried running code from the vignette manually until the error at lines 145-151 when it callsrunAPL, which calls basilisk to create the environment and near the end, I got this: > > Successfully installed numpy-2.1.2 nvidia-cublas-cu12-12.4.5.8 nvidia-cuda-cupti-cu12-12.4.127 nvidia-cuda-nvrtc-cu12-12.4.127 nvidia-cuda-runtime-cu12-12.4.127 nvidia-cudnn-cu12-9.1.0.70 nvidia-cufft-cu12-11.2.1.3 nvidia-curand-cu12-10.3.5.147 nvidia-cusolver-cu12-11.6.1.9 nvidia-cusparse-cu12-12.3.1.170 nvidia-nccl-cu12-2.21.5 nvidia-nvjitlink-cu12-12.4.127 nvidia-nvtx-cu12-12.4.127 sympy-1.13.1 torch-2.5.0 triton-3.1.0 > Done! > Error in py_call_impl(callable, call_args$unnamed, call_args$named) : > TypeError: expected np.ndarray (got numpy.ndarray) > Run `reticulate::py_last_error()` for details. > In addition: Warning message: > In rm_zeros(obj) : > Matrix contains rows with only 0s. These rows were removed. If undesired set rm_zeros = FALSE. > > reticulate::py_last_error() > > ── Python Exception Message ───────────────────────────────────────────────────────────────────────────────────────────────────────────────── > Traceback (most recent call last): > File "/home/biocbuild/bbs-3.20-bioc/R/site-library/APL/python/python_svd.py", line 7, in svd_torch > x2 = torch.from_numpy(numpy.array(x)) > TypeError: expected np.ndarray (got numpy.ndarray) > > ── R Traceback ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── > ▆ > 1. ├─APL::runAPL(...) > 2. └─APL::runAPL(...) > 3. └─APL (local) .local(...) > 4. └─APL::run_APL(...) > 5. ├─APL::cacomp(...) > 6. └─APL::cacomp(...) > 7. └─APL:::run_cacomp(...) > 8. └─basilisk::basiliskRun(...) > 9. └─APL (local) fun(...) > 10. └─reticulate (local) svd_torch(pear_res) > 11. └─reticulate:::py_call_impl(callable, call_args$unnamed, call_args$named) > See `reticulate::py_last_error()$r_trace$full_call` for more details. > > So I wondered if maybe the environment hadn’t been created successfully runningR CMD build. Would I get a different error now? AndR CMD buildandcheckran without issue.:confused:When the cache gets reset, maybe it will have the same issue? This would be the next thing to test, no basilisk cache thenR CMD build. I have not deleted basilisk cache forAPLon nebbiolo2. You won’t see it pass on the next report but the report after that. I haven’t checkedteran2.

Clemens Kohl (13:46:07) (in thread): > Hi Jen, thanks a lot for looking into it and the work you did! I am travelling this weekend so I can’t look into it right now, but I’ll try to get on it when I come back on Sunday. > I am still struggling to reproduce the errors so I can troubleshoot, but I got different errors on the BBS containers & basilsik which are likely connected with the container itself, not the code unfortunately. But your first error suprises me, as it looks like a problem between converting the numpy array to torch. > My preferred solution to this would be to get rid of the dependency altogether, but I need some time to benchmark possible solutions.

2024-10-30

Steffen Neumann (07:58:25): > Hi, I am trying to fix xcms build issue on lconway macOS 12.7.1.

Steffen Neumann (08:00:16) (in thread): > It failshttps://bioconductor.org/checkResults/devel/bioc-LATEST/xcms/lconway-buildsrc.htmlbecause of the missing dependencyfaahKO, which dies indeed need xcms for building. So a circular dependency. But it works on the other architectures and I can’t see why the workinghttps://bioconductor.org/checkResults/3.20/data-experiment-LATEST/faahKO/is not on lconway. Thoughts ?

Lori Shepherd (09:09:51) (in thread): > we can investigate post release to see why it is not installing there if it continues

Steffen Neumann (09:17:53) (in thread): > I think the issue persists for a few weeks, don’t expect it to magically disappear

Lori Shepherd (09:18:45) (in thread): > yes. thank you for keeping track. once we get through the release we can look at the logs

Hervé Pagès (23:28:17) (in thread): > arrowreinstalled on lconway. Seehttps://stat.ethz.ch/pipermail/bioc-devel/2024-October/020695.html

2024-10-31

Andres Wokaty (12:40:15) (in thread): > I’ve reinstalledfaahKOonlconway.@Hervé Pagèsinvestigated and it appeared that that package’s installation was incomplete solibrary(faahKO)didn’t findfaahKO. Let’s check Friday’s build report to see if the issue is resolved.

Andres Wokaty (13:22:23) (in thread): > Hi@Matteo Tiberti,@Hervé Pagèsthinks this might be caused bycapabilities()["X11"]beingFALSE. Usually, it’sTRUE. I don’t think I can correct it without recompiling R; however, since we just had our release, I need to let everything build for a few cycles before reinstalling R, which will be the new R version. > > I don’t see the warning on 3.21, which is building with R 4.5:https://bioconductor.org/checkResults/3.21/bioc-LATEST/Moonlight2R/(Note, I just added the windows builder yesterday so it hasn’t installed all packages; I would disregard its errors for now.) > > I have an issuehttps://github.com/Bioconductor/BBS/issues/428to revisit it in the next week or two. Thanks for your patience.

2024-11-01

Andres Wokaty (12:10:13) (in thread): > Thanks to@Hervé PagèsI didn’t need to reinstall R. I think the warning may be resolved. Please check the build report this weekend to see if it is cleared up.

Matteo Tiberti (12:17:11) (in thread): > thank you@Andres Wokatyand@Hervé Pagès! I’ll keep an eye on it

2024-11-04

Steffen Neumann (02:05:13) (in thread): > Hm,https://bioconductor.org/checkResults/3.20/bioc-LATEST/xcms/lconway-buildsrc.htmlwith Snapshot Date: 2024-11-01 13:40 -0400 still has an error due to ‘there is no package called ’faahKO’’.

Andres Wokaty (11:40:43) (in thread): > faahKOstill isn’t getting (re)installed correctly. I’ll continue to look into it.

Andres Wokaty (11:49:05) (in thread): > I didR CMD INSTALL faahKOonlconway. We’ll see if it fails again on the report tomorrow.

Steffen Neumann (11:58:36) (in thread): > So library(faahKO) succeeds now on that machine?

Andres Wokaty (11:59:31) (in thread): > yes > > lconway:~ biocbuild$ R > > R version 4.4.1 (2024-06-14) -- "Race for Your Life" > Copyright (C) 2024 The R Foundation for Statistical Computing > Platform: x86_64-apple-darwin20 > > R is free software and comes with ABSOLUTELY NO WARRANTY. > You are welcome to redistribute it under certain conditions. > Type 'license()' or 'licence()' for distribution details. > > Natural language support but running in an English locale > > R is a collaborative project with many contributors. > Type 'contributors()' for more information and > 'citation()' on how to cite R or R packages in publications. > > Type 'demo()' for some demos, 'help()' for on-line help, or > 'help.start()' for an HTML browser interface to help. > Type 'q()' to quit R. > > > library(faahKO) > Loading required package: xcms > Loading required package: BiocParallel > > This is xcms version 4.4.0 > > > Attaching package: 'xcms' > > The following object is masked from 'package:stats': > > sigma > > > >

2024-11-05

Matteo Tiberti (04:34:12) (in thread): > hi@Andres Wokatyand@Hervé Pagès, I can confirm that the packages are now all green:blush:thank you very much!

2024-11-06

Steffen Neumann (05:17:09) (in thread): > No luck, still ERROR for lconway in release, and same error on palomino7 for devel. for build with snapshot Tues 2024-11-05

2024-11-08

Andres Wokaty (10:32:44) (in thread): > After some digging, I found thatRisa’sfailureonlconwaywas alteringfaahKO. Since it’s deprecated, I added a.BBSoptionstoRisato make it unsupported on mac, so that it won’t attempt to build there. The error onpalomino7has gone away. We’ll see if this truly resolves the problem. > > On a side note, we also looked atfaahKOwhen trying to understand the problem and happened to notice that the.zzz.Rfile could use some improvement as it’s not good practice to calllibrary()andrequire()in.onLoad, which is mentioned in theNOTEinR CMD check.

2024-11-13

Steffen Neumann (10:06:44) (in thread): > Ok, I now pushed 1.47.1 using NAMESPACE/ImportFrom and removed the require(). Let’s see if that helped

2024-11-18

Steffen Neumann (04:35:46) (in thread): > So, faahKO is green in release and devel (and devel has the newNAMESPACEstuff that avoidrequire(xcms)in zzz.R):https://bioconductor.org/checkResults/release/data-experiment-LATEST/faahKO/https://bioconductor.org/checkResults/devel/data-experiment-LATEST/faahKO/RELEASE_3_20 still fails on MacOS building all three vignettes that need faahKOhttps://github.com/sneumann/xcms/blob/d3e8acffb2c8ba203702b114ad3c1116e556ee91/vignettes/xcms.Rmd#L33withthere is no package called 'faahKO', all other architectures are fine. > Shall we move this to the -devel mailling list ?

Andres Wokaty (10:23:33) (in thread): > The.BBSoptionhad not taken effect on the mac because I forgot to bumpRisa’s version, so it had still been corruptingfaahKOthere. I’ve bumped the version and reinstalledfaahKO.

2024-11-19

Andres Wokaty (12:58:12) (in thread): > Risais appearing as unsupported for macs now andxcmsis passing on all release platforms.xcmsis not passing on the linux devel builder due to strict headers for R 4.5. The error messages hint at the possible solutions.

Steffen Neumann (13:15:45) (in thread): > The header issue I can fix in the next days. Thanks for helping with the rest

2024-11-21

Hervé Pagès (16:46:45) (in thread): > If that helps,xcmsis part of the unofficial “rapid builds”:https://bioconductor.org/checkResults/3.21/bioc-rapid-LATEST/

2024-12-02

Y-h. Taguchi (02:53:16): > enrichR issue

Y-h. Taguchi (02:58:37) (in thread): > I am a maintainer of TDbasedUFEadv which now has build problem because enrichR in CRAN vanishes. I am now trying to make contact with the maintainer of enrichR. Please do not ban TDbasedUFEadv for a while.

Y-h. Taguchi (02:59:04) (in thread): > https://cran-archive.r-project.org/web/checks/2024/2024-11-04_check_results_enrichR.html

Y-h. Taguchi (03:06:06) (in thread): > https://github.com/wjawaid/enrichR/issues/87Here is a issue, and I hope that enrichR will soon come back to CRAN - Attachment: #87 enrichR removed from Cran > Hi, > > I’m not sure if you were aware, but the package has been removed from Cran and the webpage points to the following issues: > > > Package ‘enrichR’ was removed from the CRAN repository. > > > > Formerly available versions can be obtained from the archive. > > > > Archived on 2024-11-04 for policy violation. > > > > A summary of the most recent check results can be obtained from the check results archive. > > > > Please use the canonical form https://cran.r-project.org/package=enrichR to link to this page. > > Here’s a screenshot of the check results page as of 20 November 2024 > > CleanShot 2024-11-20 at 11 22 44@2x

Lori Shepherd (07:26:51) (in thread): > We can give a little more time but eventually if enrichR is not reinstated you will have to make changes.

Y-h. Taguchi (07:29:59) (in thread): > In the case that the maintainer of enrichR does not do anything, I will take over it. What is your opinion?

Lori Shepherd (07:31:47) (in thread): > If that is allowed by CRAN and you can get it re-instated that would be sufficient

Y-h. Taguchi (07:38:18) (in thread): > https://github.com/wjawaid/enrichR/pull/84“wjawaidmerged commita7bc48finto wjawaid:master17 hours ago” Thus, it will soon come back to CRAN

Lluís Revilla (08:57:34) (in thread): > Merged, doesn’t mean submitted (it has not been submitted yet according tohttps://r-hub.github.io/cransays/articles/dashboard.html) and that doesn’t mean accepted. As the package was archived for policy violation on 2024-11-04 it might need more improvements than just that PR.

Lluís Revilla (08:57:50) (in thread): > To take over a package from CRAN usually one would usually need confirmation from the previous maintainer.

Y-h. Taguchi (17:25:41) (in thread): > https://github.com/wjawaid/enrichR/pull/84#issuecomment-2512738220“I plan to give the newly updated package in the master branch a full test later in the year and possibly update the vignettes when required.” Is it too late? - Attachment: Comment on #84 Update links to ggplot function and PMC citation > Hi @wjawaid Thanks for checking in and merging all the PRs:+1:
> I plan to give the newly updated package in the master branch a full test later in the year and possibly update the vignettes when required.

2024-12-03

Lluís Revilla (02:17:03) (in thread): > It might be, better be prepared than have your packages archived from Bioconductor.

Y-h. Taguchi (02:18:21) (in thread): > You mean that you are willing to archive mine?

Lluís Revilla (02:22:26) (in thread): > I am not a Bioconductor core member, but the rules say that you cannot depend on a package that is not on CRAN or Bioconductor. There is some time as usually archiving happens close to Bioconductor releases. But I would recommend to not wait till the last moment, your users can’t install the packages from Bioconductor as there is a dependency missing from CRAN, so you are “losing” them.

Y-h. Taguchi (05:18:12) (in thread): > @Lluís RevillaI understand your point. If enrichR is permanently removed from CRAN, I will remove enrichR from mine, too. But it will soon come back to CRAN. It is a bit confusing that there are two versions with and without enrichR for single package. Anyway, he will submit the updated version till 15th Dec.https://github.com/wjawaid/enrichR/pull/84#issuecomment-2513931501 - Attachment: Comment on #84 Update links to ggplot function and PMC citation > Hi. Thank you very much for your contributions. I plan to submit it again
> around 15 December. I just want to check it is working properly. > > On Mon, 2 Dec 2024, 22:30 Y-h. Taguchi, @*.******> wrote: > > > Is it possible for you to tell me when it is planned to submit to CRAN? I
> > am using enrichR in my bioconductor package, TDbasedUFEadv, and
> > Biocouductor forces me to remove it from my package….. > > > > —
> > Reply to this email directly, view it on GitHub
> > <https://github.com/wjawaid/enrichR/pull/84#issuecomment-2513098252|#84 (comment)>, or
> > unsubscribe
> > https://github.com/notifications/unsubscribe-auth/ABPZD7SPNDUBEA3YT7ECN532DTNOZAVCNFSM6AAAAABRJVI4NWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKMJTGA4TQMRVGI
> > .
> > You are receiving this because you were mentioned.Message ID:
> > @*.******>

2024-12-06

Vince Carey (18:27:33) (in thread): > I had a linkedin message from enrichR author who said he is preparing to resubmit.

2024-12-09

Anthony Christidis (14:17:39): > @Anthony Christidis has joined the channel

Anthony Christidis (14:20:12): > Hello, my packagescDiagnosticshas been on Bioconductor for a few months now, and it came to my attention (credit to@Ludwig Geistlinger) that the download stats for the package seems to not display at all and return an error:https://bioconductor.org/packages/stats/bioc/scDiagnostics/Any help to fix this issue as fast as possible would be highly appreciated.

Lori Shepherd (14:25:33): > Thank you for the notification – I will investigate this immediately

Anthony Christidis (15:51:52) (in thread): > Thanks, Lori!

2024-12-10

Lori Shepherd (13:24:04): > This should be resolved. sorry for the inconvenience

Anthony Christidis (18:33:50) (in thread): > Thanks, Lori.

2024-12-19

Sergio Oller (17:17:11): > @Sergio Oller has joined the channel

2024-12-21

Sergio Oller (05:27:24): > Hi, I am the maintainer of the MassSpecWavelet Bioconductor package.https://www.bioconductor.org/packages/release/bioc/html/MassSpecWavelet.htmlOn December 18th I pushed a fix to an off-by-one error on a C function in the package, that was causing a crash:https://code.bioconductor.org/browse/MassSpecWavelet/commits/RELEASE_3_20The fix was pushed to the devel branch and the RELEASE_3_20 branch. Both branches had their respective NEWS entry and a patch version number increase. > > After more than 48 hours, the build report for the stable release has not been updatedhttps://bioconductor.org/checkResults/3.20/bioc-LATEST/MassSpecWavelet/The devel version however has been updated and the fix version has been released:https://bioconductor.org/checkResults/3.21/bioc-LATEST/MassSpecWavelet/Is there anything else I have to do to get the stable version built and released? > > CRAN package maintainers depending on this fix are getting nervous :-( > > Thanks for any help and happy holidays/end of the year if you are celebrating it!

Hervé Pagès (14:41:43) (in thread): > We only build the release branch twice a week. New reports on Mondays and Thursdays, as indicated here:https://bioconductor.org/checkResults/

2024-12-22

Aaron Lun (15:22:05): > bioconductor/bioconductor_docker:develDocker image no longer haslinux/amd64option. pretty sure that used to be around; currently causing failures on GitHub CI, e.g.,https://github.com/SingleR-inc/SingleR/actions/runs/12438109528/job/34732333147

2024-12-23

Vince Carey (09:12:05): > @Alex Mahmoud^^

Alex Mahmoud (09:15:58): > I’lllook into it, this is likely a bug with the new building GHA that builds the twoarch’sseparately on native compute then merges the two platform images, instead of using emulator. Thanks@Aaron Lunfor reporting!!

Y-h. Taguchi (16:28:13) (in thread): > He is still working on it.https://github.com/wjawaid/enrichR/issues/87#issuecomment-2560311382 - Attachment: Comment on #87 enrichR removed from Cran > Yes the is the plan. Am working on it. > > > On 23 Dec 2024, at 21:12, Y-h. Taguchi @*.******> wrote: > > > > @wjawaid https://github.com/wjawaid I am glad if you can tell me when you are willing to re-submit it to CRAN. > > > > —
> > Reply to this email directly, view it on GitHub <https://github.com/wjawaid/enrichR/issues/87#issuecomment-2560302362|#87 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABPZD7R423HLAWAV2JGJDCD2HB4E5AVCNFSM6AAAAABSFKOBAWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKNRQGMYDEMZWGI.
> > You are receiving this because you were mentioned.

2024-12-28

apekshya kandel (07:54:18): > @apekshya kandel has joined the channel

Pascal-Onaho (07:55:08): > @Pascal-Onaho has joined the channel

2024-12-29

Yahya Jahun (04:01:30): > @Yahya Jahun has joined the channel

2025-01-02

Najuma Najeem (10:59:57): > @Najuma Najeem has joined the channel

2025-01-08

Peter Hickey (17:00:56): > There don’t seem to have been any software builds for BioC 3.20 since 2025-01-02 and 3.21 since 2025-01-04?

Lori Shepherd (20:00:24): > Sorry there was not an announcement.There was maintenance being done on central builders that finished today. We expect normal builds here on out

Peter Hickey (20:10:04): > Thanks, Lori!

2025-01-09

Ammar Sabir Cheema (11:29:14): > @Ammar Sabir Cheema has joined the channel

2025-01-17

Julian Stamp (10:01:40): > @Julian Stamp has joined the channel

2025-01-18

Wanling Kratzman (09:07:16): > @Wanling Kratzman has joined the channel

2025-01-21

Martin Grigorov (06:27:30): > Hi- only two builders are listed in the index page (nebbiolo2andpalomino5).merida1,kjohnson1andtaishanare missing today

Andres Wokaty (10:34:57) (in thread): > I noticed over the weekend there was an issue at the start of the build so only those two machines were able to complete building. All machines will be on the next report later in the week.

2025-01-24

Aidan Lakshman (14:16:56): > Seeing a weird error on OSX builds for Bioc, can’t reproduce them locally:https://bioconductor.org/checkResults/devel/bioc-LATEST/SynExtend/kjohnson3-install.htmlrelevant error seems to beld: library 'emutls_w' not foundjust want to check if this is a build-side error before I jump down this rabbit hole. I don’t have anything crazy in myMakevarsfile, but happy to share in case that makes a difference.

Kasper D. Hansen (14:19:57): > I think the devel builders for macOS are still being configured. We are trying to match what they do at CRAN, but I don’t think the CRAN setup has been finalized. So for now, I would wait.

Kasper D. Hansen (14:20:35): > Having said that, there are plenty of people who know more about the current status than me.

Aidan Lakshman (14:25:51): > That’s what I figured, just wanted to double check! Thanks for the quick response, I’ll keep an eye on it to see if it resolves itself or if I need to fix something down the line.

Kasper D. Hansen (14:31:30): > It might take as much as a few weeks. It might make sense to have a post about this on Bioc-devel (and here) when the setup is working (aka when any errors should be looked at by the developers)

Vince Carey (14:33:07): > Version of gfortran used to build R for mac has changed recently. It will be resolved for our builds soon.

Kasper D. Hansen (14:34:41): > CRAN is still not posting R-devel binaries for macOS, so something is not finalized on their end.

Lluís Revilla (14:39:37): > Is anyone in contact with the CRAN team about this? I can reach out to them/Simon Urbanek if needed

Kasper D. Hansen (14:53:51): > We know Simon is working on this. He has previously predicted it should be working by now. I am not sure he needs any additional encouragement.

Lori Shepherd (16:11:38): > Agreed. We know he is working on it so I don’t think there is any need too reach out more

2025-01-27

Andres Wokaty (12:29:33): > Today’s devel report reflects the adjustment to gfortran 14.2.0 for the macs. Let me know if you notice or run into any issues.

2025-01-28

Aidan Lakshman (10:10:47): > my builds are now passing, I checked yesterday. Thanks again for the quick responses!

2025-02-17

Davide Risso (02:57:38): > Cross-posting here in case people can help - Attachment: Attachment > Hi all, I hope this is the right channel. My scone package is failing in devel and I’m trying to reproduce the error in the bioconductor devel Docker. However, when trying to installing beachmat (an indirect dependency of scone). I get the following error: > > BiocManager::install("beachmat") > 'getOption("repos")' replaces Bioconductor standard repositories, see 'help("repositories", package = "BiocManager")' for > details. > Replacement repositories: > CRAN: [https://cloud.r-project.org](https://cloud.r-project.org) > Bioconductor version 3.21 (BiocManager 1.30.25), R Under development (unstable) (2025-02-12 r87715) > Installing package(s) 'beachmat' > trying URL '[https://bioconductor.org/packages/3.21/bioc/src/contrib/beachmat_2.23.6.tar.gz](https://bioconductor.org/packages/3.21/bioc/src/contrib/beachmat_2.23.6.tar.gz)' > Content type 'application/x-gzip' length 377367 bytes (368 KB) > ================================================== > downloaded 368 KB > > * installing **source** package 'beachmat' ... > **** this is package 'beachmat' version '2.23.6' > **** using staged installation > **** libs > using C++ compiler: 'g++ (Ubuntu 13.3.0-6ubuntu2~24.04) 13.3.0' > using C++17 > g++ -std=gnu++17 -I"/usr/local/lib/R/include" -DNDEBUG -I../inst/include -I'/usr/local/lib/R/site-library/Rcpp/include' -I'/usr/local/lib/R/host-site-library/assorthead/include' -I/usr/local/include -fpic -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g -c RcppExports.cpp -o RcppExports.o > g++ -std=gnu++17 -I"/usr/local/lib/R/include" -DNDEBUG -I../inst/include -I'/usr/local/lib/R/site-library/Rcpp/include' -I'/usr/local/lib/R/host-site-library/assorthead/include' -I/usr/local/include -fpic -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g -c constant_matrix.cpp -o constant_matrix.o > g++ -std=gnu++17 -I"/usr/local/lib/R/include" -DNDEBUG -I../inst/include -I'/usr/local/lib/R/site-library/Rcpp/include' -I'/usr/local/lib/R/host-site-library/assorthead/include' -I/usr/local/include -fpic -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g -c delayed_isometric_binary.cpp -o delayed_isometric_binary.o > g++ -std=gnu++17 -I"/usr/local/lib/R/include" -DNDEBUG -I../inst/include -I'/usr/local/lib/R/site-library/Rcpp/include' -I'/usr/local/lib/R/host-site-library/assorthead/include' -I/usr/local/include -fpic -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g -c delayed_isometric_math.cpp -o delayed_isometric_math.o > g++ -std=gnu++17 -I"/usr/local/lib/R/include" -DNDEBUG -I../inst/include -I'/usr/local/lib/R/site-library/Rcpp/include' -I'/usr/local/lib/R/host-site-library/assorthead/include' -I/usr/local/include -fpic -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g -c delayed_isometric_unary.cpp -o delayed_isometric_unary.o > g++: fatal error: Killed signal terminated program cc1plus > compilation terminated. > make: ***** [/usr/local/lib/R/etc/Makeconf:210: delayed_isometric_unary.o] Error 1 > ERROR: compilation failed for package 'beachmat' > * removing '/usr/local/lib/R/host-site-library/beachmat' > Any idea of what’s going on here? Any help is much appreciated!

Davide Risso (09:32:48) (in thread): > Solved in#containers

2025-02-18

Hervé Pagès (16:28:52) (in thread): > oops, looks like the error is caused by a recent change inDelayedArray(this onehttps://github.com/Bioconductor/DelayedArray/commit/ce63724c3aac4528d229ee6e49232472667553d5). Let me take a look.

Hervé Pagès (16:32:00) (in thread): > Here’s a MRE: > > library(DelayedArray) > library(scone) > x <- DelayedArray(poissonSparseMatrix(10, 8)) > scone:::computePsiNormSF(x) >

2025-02-19

Hervé Pagès (02:20:34) (in thread): > @Davide RissoI submitted a PR:https://github.com/YosefLab/scone/pull/108 - Attachment: #108 Repair and optimize PsiNorm() > Hi, > A recent change in DelayedArray (in version 0.33.5) broke PsiNorm() on a sparse DelayedMatrix object and on a SE/SCE object with counts represented as a sparse DelayedMatrix object. See https://bioconductor.org/checkResults/3.21/bioc-LATEST/scone/. Sorry for that! > This commit fixes the problem and also introduces two small improvements to PsiNorm(): > • makes PsiNorm() work on a SparseMatrix object; > • makes PsiNorm() slightly faster by reducing the number of matrix transpositions from 4 to 2. > For example: > > > library(scone) > > library(DelayedArray) > x <- DelayedArray(poissonSparseMatrix(10, 8)) > # Broken in scone 1.31.0 (due to recent change in DelayedArray) but works again in scone 1.31.1 > PsiNorm(x) > > library(TENxPBMCData) > sce <- TENxPBMCData("pbmc4k") > # Also broken in scone 1.31.0 (due to recent change in DelayedArray) but works again in scone 1.31.1 > PsiNorm(sce) > > > > Other improvements: > > > x0 <- poissonSparseMatrix(5000, 77000) > a0 <- as.matrix(x0) > > PsiNorm(a0) # 50% speedup compared to scone 1.30.0 > > PsiNorm(x0) # NEW (not supported in scone 1.30.0), 10x faster than 'PsiNorm(a0)' > > sce <- SingleCellExperiment(list(counts=x0)) > PsiNorm(sce) # NEW (not supported in scone 1.30.0) > > > > Best,
> H.

Davide Risso (06:41:06) (in thread): > Thanks@Hervé Pagès!!

Anushka Paharia (14:06:46): > @Anushka Paharia has joined the channel

2025-02-25

Michael Love (12:08:10): > I just sawDESeq2had build errors across the board in devel bc missingpasilla, any idea what that was about?https://bioconductor.org/checkResults/devel/data-experiment-LATEST/pasilla/

Lori Shepherd (12:18:28): > It looks like its a circular dependency –@Andres Wokatywas R updated recently late last week so the repos reset (and possibly why experimentdata didn’t build last thur?) Cause it theoretically should clear up on the next build since pasilla installed but wouldn’t propagate because DEXSeq is not available – and DEXSeq failed because pasilla is not available – but on the next round since pasilla installed DEXSeq should find it and then in turn the next experimentdata build pasilla should find DEXSeq — theoretically …

Andres Wokaty (12:21:41): > yes, I updated R devel on nebbiolo1 last Thursday and stopped the experiment builds.

Lori Shepherd (12:23:58): > so we should keep an eye on this and make sure it clears up

Andres Wokaty (12:24:20): > They should finish around 3pm ET today

Kasper D. Hansen (12:35:33): > Today I worked on Rgraphviz and updated it from 2.51.0 to 2.51.1. I finished this a little around 11:45-12. I have confirmed that this is reflected on the bioc git server . I also renamed the default branch from master to devel. About this time, I got a build report saying 2.51.0 has build failures. Was this just a coincidence? (That I get a build report right at this time) or has the system somehow change to trigger when something happens?

Lori Shepherd (12:36:21): > just coincidence.

Lori Shepherd (14:39:45) (in thread): > although looking back through my notes and on the landing page DEXSeq has been having issues since the beginning of the last release. I also reached out in Dec to make them aware of the requested deprecation of a dependent package. The maintainer has never responded so it may be more complicated even after pasilla becomes available….

Michael Love (15:15:20) (in thread): > @Wolfgang Huberpinging about DEXSeq

Wolfgang Huber (15:15:31): > @Wolfgang Huber has joined the channel

2025-02-26

Hervé Pagès (19:26:03) (in thread): > yep,DEXSeqhas been in ERROR for months for reasons unrelated topasilla(seehttps://bioconductor.org/checkResults/3.21/bioc-LATEST/DEXSeq/) so the latter was never allowed to propagate to the BioC 3.21 repos

Michael Love (20:58:32) (in thread): > Ok I’ll ask Wolfgang or Simon to take a look

2025-02-27

Lori Shepherd (07:17:35) (in thread): > If you talk to them it might be good to update the maintainer email if Alejandro is not going to stay on top of it

2025-03-03

Michael Love (14:07:51): > Re: DESeq2 — i’m going to resolve this today hopefully, by just cutting off DESeq2 from pasilla package dependency

2025-03-04

Kasper D. Hansen (14:19:51): > On the windows build servers, is it possible to see what version of Rtools is installed?

Kasper D. Hansen (14:20:01): > I don’t see it herehttps://bioconductor.org/checkResults/devel/bioc-LATEST/palomino7-NodeInfo.html

Kasper D. Hansen (14:20:31): > (perhaps there is a 1-1 correspondence between Rtools and GCC version; I am not sure)

Andres Wokaty (21:01:28): > Both machines have Rtools44 and they use the gcc available in it, which is listed on that page. Rtools updates are general infrequent and Tomas is great about telling us when a new one is available, so it’s usually whatever is availablehttps://cran.r-project.org/bin/windows/Rtools/rtools44/rtools.html. I usually also document the version onhttps://github.com/Bioconductor/BBS/blob/devel/Doc/Prepare-Windows-Server-2022-HOWTO.md#25-install-rtools. - File (PNG): image.png

Kasper D. Hansen (21:08:59): > Thanks.I got an email from Tomas about a release candidate for a new version which causes some issues with Rgraphviz.That’s whyI’masking.

2025-03-05

Lori Shepherd (07:25:40) (in thread): > It looks like recent changes to RgraphViz might have also broken in devel; not sure if it is the same or related but also something to look into sooner than later as it is affecting many packages > > library(graph) > g <- graphNEL(nodes="A", edgeL=list(A=list(edges="A"))) > library(Rgraphviz) > plot(g) > ***** buffer overflow detected *****: terminated > Aborted (core dumped) >

Kasper D. Hansen (09:04:44) (in thread): > Yeah, I see this and I am pretty baffled. I don’t see why the recent changes would result in this at all. So frankly, I am hoping it is a fluke.

Lori Shepherd (09:08:35) (in thread): > I have a script to keep track of failing packages to warn for deprecation; per my records Rgraphviz has been failing in devel since Feb 24 daily…

Lori Shepherd (09:38:59) (in thread): > The circular dependency issue cleared up but unfortunately because DEXSeq is still broken pasilla is not propagating. I just emailed the maintainer and author again.

Michael Love (09:45:05) (in thread): > got it. hopefully DESeq2 1.47.4 will not be affected

Michael Love (09:45:11) (in thread): > i think it will show up today

Michael Love (13:38:18) (in thread): > oops i left in one more link to pasilla in the vignette, just fixed in 1.47.5

Michael Love (13:38:31) (in thread): > oops i left in one more reference to pasilla in the vignette, just fixed in 1.47.5

Lori Shepherd (15:40:06): > Bioconductor 3.21 Release Schedule:https://bioconductor.org/developers/release-schedule/

2025-03-06

Michael Love (12:54:28) (in thread): - File (PNG): Screenshot 2025-03-06 at 12.54.00 PM.png

2025-03-07

Lori Shepherd (12:16:09) (in thread): > @Kasper D. Hansenjust want to make sure this stays on your radar as mentioned doesn’t seem like a fluke – and also the bioc-devel mailing list email that came through today with an additional error seen in a few packages.

Kasper D. Hansen (13:04:17) (in thread): > It’son my radar.I’mvisiting Boston yesterday and today but I hope to get hacking over the weekend.

2025-03-13

Mihai Todor (06:59:42): > @Mihai Todor has joined the channel

Lori Shepherd (14:33:45) (in thread): > If possible maybe mention on the mailing list that you are looking into it?https://stat.ethz.ch/pipermail/bioc-devel/2025-March/020878.html

2025-03-31

Eva Hamrud (19:21:17): > @Eva Hamrud has joined the channel

2025-04-13

Fanding Zhou (19:52:03): > @Fanding Zhou has joined the channel

2025-04-16

Lori Shepherd (15:21:10): > Bioconductor 3.21 is released! > Thanks to all developers and community members for contributing to the project! > Please see the full release announcement:https://bioconductor.org/news/bioc_3_21_release/ - Attachment (bioconductor.org): Bioconductor - Bioconductor 3.21 Released > The Bioconductor project aims to develop and share open source software for precise and repeatable analysis of biological data. We foster an inclusive and collaborative community of developers and data scientists.

2025-04-18

Jeroen Ooms (05:21:20): > Why are some packages in the registry missing thegit_last_commit_datefield? > > bioc <- jsonlite::read_json('[https://bioconductor.org/packages/json/3.22/bioc/packages.json](https://bioconductor.org/packages/json/3.22/bioc/packages.json)') > nodate <- Filter(function(x) {is.null(x$git_last_commit_date)}, bioc) > length(nodate) > > I am interested in this date to know which packages need to be updated.

Lori Shepherd (06:26:29) (in thread): > We’lllook into it. Thanks for bringing it to our attention

Lori Shepherd (08:22:07) (in thread): > So at I think will smooth out over the next week post release – when I look at the 3.21 version the only ones that are missing a git_last_commit_date are ones that are deprecated that have not built – just looking at the last 3 or 4 packages in that list, they may active/not deprecated but they have not propagated because of other issues – The VIEWS file (that the referenced json is built off of) I think is created based on propagated packages for full information, and then these packages that have not built/checked/propagated are included but with limited information as they are technically not distributed yet (I think)

Lori Shepherd (08:28:58) (in thread): > I’m not sure when the git_last_commit date is added to the VIEWS file per package but maybe we can take a look to see if this information can still be added for these types of packages (I’m not sure how it is keeping track or pulls this info so it may or may not be possible@Andres Wokaty)

Lori Shepherd (11:03:22) (in thread): > https://github.com/Bioconductor/biocViews/issues/21 - Attachment: #21 Missing git_last_commit information > Sort of related to #14 > > Sparked from conversation: https://community-bioc.slack.com/archives/CEQ04GKEC/p1744968080782439 > > So for packages that haven’t propagated certain information is missing. Can we extract more information for these types of packages despite not propagating? > > example.
> git_last_commit information is added to DESCRIPTION by BBS so that biocViews repository.R can use default tools:::.build_repository_package_db to extract it in a nice format. The BBS extracts this information from a gitlog_file. Can we use this file to get the git information for packages not propagated.

Jeroen Ooms (11:07:35) (in thread): > OK thanks for looking into it!

Lori Shepherd (14:34:50) (in thread): > For What Its Worth Tho: The git_last_commit_date from our perspective is the commit related to the propagated package ( so that builds/checks and able to propagate) so that the end user can trace back the commit that is distributed to them. So for all intensive purposes skipping ones that don’t have a git_last_commit_date would be consistent with what is currently propagated to end users through Bioconductor official avenues.

2025-04-19

Jeroen Ooms (07:06:24) (in thread): > OK I assumed it was the current HEAD commit on the git server. Can you remind me, what is the delay for a package commit to get propagated? Does it happen nightly?

Lori Shepherd (10:00:59) (in thread): > No. It is the commit of the version that propagated to users. > Software packages are built once a day. We do a pull of all packages I think around 1 pm ET. So it can be anywhere from 24-48 depending on when the commit occurs; and it must pass our build/check to propagate too.

2025-04-30

MARTA SEVILLA PORRAS (12:13:58): > @MARTA SEVILLA PORRAS has joined the channel